Note: This is test shard 2 of 8.
[==========] Running 8 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from RebalanceIgnoredTserversTest
[ RUN ] RebalanceIgnoredTserversTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 21:25:21.748715 25482 test_util.cc:268] Using random seed: 1033199300
I20240502 21:25:21.759016 25482 ts_itest-base.cc:112] Starting cluster with:
I20240502 21:25:21.759083 25482 ts_itest-base.cc:113] --------------
I20240502 21:25:21.759124 25482 ts_itest-base.cc:114] 5 tablet servers
I20240502 21:25:21.759159 25482 ts_itest-base.cc:115] 3 replicas per TS
I20240502 21:25:21.759199 25482 ts_itest-base.cc:116] --------------
2024-05-02T21:25:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T21:25:21Z Disabled control of system clock
I20240502 21:25:21.781387 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.226.190:36021
--webserver_interface=127.24.226.190
--webserver_port=0
--builtin_ntp_servers=127.24.226.148:35721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.226.190:36021
--enable_range_replica_placement=false with env {}
W20240502 21:25:22.125866 25494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:22.126153 25494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:22.126283 25494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:22.135664 25494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 21:25:22.135789 25494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:22.135839 25494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 21:25:22.135891 25494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 21:25:22.145491 25494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:35721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.24.226.190:36021
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.226.190:36021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.24.226.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:22.146651 25494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:22.148496 25494 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:22.159930 25500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:22.160425 25499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:22.160495 25502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:22.161473 25494 server_base.cc:1034] running on GCE node
I20240502 21:25:22.162153 25494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:22.163661 25494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:22.164871 25494 hybrid_clock.cc:648] HybridClock initialized: now 1714685122164786 us; error 41 us; skew 500 ppm
I20240502 21:25:22.165165 25494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:22.171599 25494 webserver.cc:460] Webserver started at http://127.24.226.190:46251/ using document root <none> and password file <none>
I20240502 21:25:22.172252 25494 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:22.172372 25494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:22.172688 25494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:22.175225 25494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "ad48e1e99edd4e6e820d0ed3f632ab90"
format_stamp: "Formatted at 2024-05-02 21:25:22 on dist-test-slave-41k1"
I20240502 21:25:22.175843 25494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "ad48e1e99edd4e6e820d0ed3f632ab90"
format_stamp: "Formatted at 2024-05-02 21:25:22 on dist-test-slave-41k1"
I20240502 21:25:22.180585 25494 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20240502 21:25:22.183668 25508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:22.184878 25494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:25:22.185031 25494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ad48e1e99edd4e6e820d0ed3f632ab90"
format_stamp: "Formatted at 2024-05-02 21:25:22 on dist-test-slave-41k1"
I20240502 21:25:22.185242 25494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:22.198411 25494 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:22.199016 25494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:22.199275 25494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:22.221645 25494 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.190:36021
I20240502 21:25:22.221678 25559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.190:36021 every 8 connection(s)
I20240502 21:25:22.222844 25494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 21:25:22.224872 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 25494
I20240502 21:25:22.225149 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 21:25:22.227049 25560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:22.236513 25560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad48e1e99edd4e6e820d0ed3f632ab90: Bootstrap starting.
I20240502 21:25:22.239051 25560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad48e1e99edd4e6e820d0ed3f632ab90: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:22.240190 25560 log.cc:824] T 00000000000000000000000000000000 P ad48e1e99edd4e6e820d0ed3f632ab90: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:22.242944 25560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad48e1e99edd4e6e820d0ed3f632ab90: No bootstrap required, opened a new log
I20240502 21:25:22.249421 25560 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad48e1e99edd4e6e820d0ed3f632ab90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad48e1e99edd4e6e820d0ed3f632ab90" member_type: VOTER last_known_addr { host: "127.24.226.190" port: 36021 } }
I20240502 21:25:22.249778 25560 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad48e1e99edd4e6e820d0ed3f632ab90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:22.249907 25560 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad48e1e99edd4e6e820d0ed3f632ab90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad48e1e99edd4e6e820d0ed3f632ab90, State: Initialized, Role: FOLLOWER
I20240502 21:25:22.250545 25560 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad48e1e99edd4e6e820d0ed3f632ab90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad48e1e99edd4e6e820d0ed3f632ab90" member_type: VOTER last_known_addr { host: "127.24.226.190" port: 36021 } }
I20240502 21:25:22.250787 25560 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad48e1e99edd4e6e820d0ed3f632ab90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:22.250906 25560 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad48e1e99edd4e6e820d0ed3f632ab90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:22.251051 25560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad48e1e99edd4e6e820d0ed3f632ab90 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:22.252988 25560 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad48e1e99edd4e6e820d0ed3f632ab90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad48e1e99edd4e6e820d0ed3f632ab90" member_type: VOTER last_known_addr { host: "127.24.226.190" port: 36021 } }
I20240502 21:25:22.253428 25560 leader_election.cc:302] T 00000000000000000000000000000000 P ad48e1e99edd4e6e820d0ed3f632ab90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad48e1e99edd4e6e820d0ed3f632ab90; no voters:
I20240502 21:25:22.253971 25560 leader_election.cc:288] T 00000000000000000000000000000000 P ad48e1e99edd4e6e820d0ed3f632ab90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:22.254082 25566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad48e1e99edd4e6e820d0ed3f632ab90 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:22.254623 25566 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad48e1e99edd4e6e820d0ed3f632ab90 [term 1 LEADER]: Becoming Leader. State: Replica: ad48e1e99edd4e6e820d0ed3f632ab90, State: Running, Role: LEADER
I20240502 21:25:22.255136 25566 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad48e1e99edd4e6e820d0ed3f632ab90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad48e1e99edd4e6e820d0ed3f632ab90" member_type: VOTER last_known_addr { host: "127.24.226.190" port: 36021 } }
I20240502 21:25:22.255609 25560 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad48e1e99edd4e6e820d0ed3f632ab90 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:25:22.257989 25568 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad48e1e99edd4e6e820d0ed3f632ab90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad48e1e99edd4e6e820d0ed3f632ab90. Latest consensus state: current_term: 1 leader_uuid: "ad48e1e99edd4e6e820d0ed3f632ab90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad48e1e99edd4e6e820d0ed3f632ab90" member_type: VOTER last_known_addr { host: "127.24.226.190" port: 36021 } } }
I20240502 21:25:22.257958 25567 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad48e1e99edd4e6e820d0ed3f632ab90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad48e1e99edd4e6e820d0ed3f632ab90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad48e1e99edd4e6e820d0ed3f632ab90" member_type: VOTER last_known_addr { host: "127.24.226.190" port: 36021 } } }
I20240502 21:25:22.258257 25568 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad48e1e99edd4e6e820d0ed3f632ab90 [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:22.258386 25567 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad48e1e99edd4e6e820d0ed3f632ab90 [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:22.259889 25572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:25:22.263933 25572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:25:22.276980 25572 catalog_manager.cc:1347] Generated new cluster ID: 31c5df81b03e4d759226ca4cce3d8e17
I20240502 21:25:22.277091 25572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:25:22.292081 25572 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:25:22.293467 25572 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:25:22.304584 25572 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ad48e1e99edd4e6e820d0ed3f632ab90: Generated new TSK 0
I20240502 21:25:22.305152 25572 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:25:22.319095 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.129:0
--local_ip_for_outbound_sockets=127.24.226.129
--webserver_interface=127.24.226.129
--webserver_port=0
--tserver_master_addrs=127.24.226.190:36021
--builtin_ntp_servers=127.24.226.148:35721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240502 21:25:22.667749 25585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:22.668013 25585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:22.668148 25585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:22.674427 25585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:22.674557 25585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.129
I20240502 21:25:22.682116 25585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:35721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.24.226.129
--webserver_port=0
--tserver_master_addrs=127.24.226.190:36021
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.129
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:22.683190 25585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:22.684679 25585 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:22.694973 25593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:22.695019 25591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:22.695000 25590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:22.695698 25585 server_base.cc:1034] running on GCE node
I20240502 21:25:22.696167 25585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:22.697031 25585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:22.698228 25585 hybrid_clock.cc:648] HybridClock initialized: now 1714685122698180 us; error 33 us; skew 500 ppm
I20240502 21:25:22.698448 25585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:22.700281 25585 webserver.cc:460] Webserver started at http://127.24.226.129:46285/ using document root <none> and password file <none>
I20240502 21:25:22.700865 25585 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:22.700959 25585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:22.701182 25585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:22.703321 25585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "3bc83d3565d14e8399843f8b0f593e53"
format_stamp: "Formatted at 2024-05-02 21:25:22 on dist-test-slave-41k1"
I20240502 21:25:22.703951 25585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "3bc83d3565d14e8399843f8b0f593e53"
format_stamp: "Formatted at 2024-05-02 21:25:22 on dist-test-slave-41k1"
I20240502 21:25:22.708225 25585 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20240502 21:25:22.711091 25599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:22.712096 25585 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:22.712220 25585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "3bc83d3565d14e8399843f8b0f593e53"
format_stamp: "Formatted at 2024-05-02 21:25:22 on dist-test-slave-41k1"
I20240502 21:25:22.712347 25585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:22.735467 25585 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:22.736042 25585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:22.736261 25585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:22.737059 25585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:22.738401 25585 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:22.738466 25585 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:22.738545 25585 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:22.738584 25585 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:22.766134 25585 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.129:39653
I20240502 21:25:22.766186 25713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.129:39653 every 8 connection(s)
I20240502 21:25:22.767267 25585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 21:25:22.774052 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 25585
I20240502 21:25:22.774235 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 21:25:22.777441 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.130:0
--local_ip_for_outbound_sockets=127.24.226.130
--webserver_interface=127.24.226.130
--webserver_port=0
--tserver_master_addrs=127.24.226.190:36021
--builtin_ntp_servers=127.24.226.148:35721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 21:25:22.778618 25714 heartbeater.cc:347] Connected to a master server at 127.24.226.190:36021
I20240502 21:25:22.778882 25714 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:22.779717 25714 heartbeater.cc:511] Master 127.24.226.190:36021 requested a full tablet report, sending...
I20240502 21:25:22.781509 25525 ts_manager.cc:194] Registered new tserver with Master: 3bc83d3565d14e8399843f8b0f593e53 (127.24.226.129:39653)
I20240502 21:25:22.782729 25525 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.129:43071
W20240502 21:25:23.074959 25718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:23.075201 25718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:23.075277 25718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:23.081939 25718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:23.082082 25718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.130
I20240502 21:25:23.090129 25718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:35721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.24.226.130
--webserver_port=0
--tserver_master_addrs=127.24.226.190:36021
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.130
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:23.091172 25718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:23.092692 25718 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:23.102835 25724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:23.102931 25726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:23.103060 25723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:23.103911 25718 server_base.cc:1034] running on GCE node
I20240502 21:25:23.104405 25718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:23.105194 25718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:23.106406 25718 hybrid_clock.cc:648] HybridClock initialized: now 1714685123106348 us; error 35 us; skew 500 ppm
I20240502 21:25:23.106668 25718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:23.108848 25718 webserver.cc:460] Webserver started at http://127.24.226.130:41585/ using document root <none> and password file <none>
I20240502 21:25:23.109519 25718 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:23.109647 25718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:23.109922 25718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:23.112166 25718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "4f5fa89cbeac47e5bf611a21252c7732"
format_stamp: "Formatted at 2024-05-02 21:25:23 on dist-test-slave-41k1"
I20240502 21:25:23.112843 25718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "4f5fa89cbeac47e5bf611a21252c7732"
format_stamp: "Formatted at 2024-05-02 21:25:23 on dist-test-slave-41k1"
I20240502 21:25:23.117208 25718 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 21:25:23.120132 25732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:23.121246 25718 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 21:25:23.121369 25718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "4f5fa89cbeac47e5bf611a21252c7732"
format_stamp: "Formatted at 2024-05-02 21:25:23 on dist-test-slave-41k1"
I20240502 21:25:23.121505 25718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:23.139169 25718 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:23.139786 25718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:23.140020 25718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:23.140785 25718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:23.141991 25718 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:23.142051 25718 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:23.142154 25718 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:23.142204 25718 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:23.169765 25718 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.130:34235
I20240502 21:25:23.169816 25844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.130:34235 every 8 connection(s)
I20240502 21:25:23.170799 25718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 21:25:23.173085 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 25718
I20240502 21:25:23.173240 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 21:25:23.175909 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.131:0
--local_ip_for_outbound_sockets=127.24.226.131
--webserver_interface=127.24.226.131
--webserver_port=0
--tserver_master_addrs=127.24.226.190:36021
--builtin_ntp_servers=127.24.226.148:35721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 21:25:23.182447 25845 heartbeater.cc:347] Connected to a master server at 127.24.226.190:36021
I20240502 21:25:23.182677 25845 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:23.183342 25845 heartbeater.cc:511] Master 127.24.226.190:36021 requested a full tablet report, sending...
I20240502 21:25:23.184523 25525 ts_manager.cc:194] Registered new tserver with Master: 4f5fa89cbeac47e5bf611a21252c7732 (127.24.226.130:34235)
I20240502 21:25:23.185163 25525 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.130:43961
W20240502 21:25:23.462950 25848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:23.463209 25848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:23.463349 25848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:23.470389 25848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:23.470556 25848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.131
I20240502 21:25:23.479219 25848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:35721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.24.226.131
--webserver_port=0
--tserver_master_addrs=127.24.226.190:36021
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.131
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:23.480209 25848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:23.481666 25848 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:23.492837 25853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:23.492909 25854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:23.493763 25856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:23.493840 25848 server_base.cc:1034] running on GCE node
I20240502 21:25:23.494346 25848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:23.495023 25848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:23.496223 25848 hybrid_clock.cc:648] HybridClock initialized: now 1714685123496168 us; error 31 us; skew 500 ppm
I20240502 21:25:23.496500 25848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:23.498696 25848 webserver.cc:460] Webserver started at http://127.24.226.131:44505/ using document root <none> and password file <none>
I20240502 21:25:23.499249 25848 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:23.499343 25848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:23.499588 25848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:23.501726 25848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "848792f66684491e974a2ff9faae212f"
format_stamp: "Formatted at 2024-05-02 21:25:23 on dist-test-slave-41k1"
I20240502 21:25:23.502317 25848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "848792f66684491e974a2ff9faae212f"
format_stamp: "Formatted at 2024-05-02 21:25:23 on dist-test-slave-41k1"
I20240502 21:25:23.506804 25848 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 21:25:23.510102 25862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:23.511193 25848 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 21:25:23.511373 25848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "848792f66684491e974a2ff9faae212f"
format_stamp: "Formatted at 2024-05-02 21:25:23 on dist-test-slave-41k1"
I20240502 21:25:23.511569 25848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:23.527299 25848 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:23.527957 25848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:23.528228 25848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:23.529127 25848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:23.530346 25848 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:23.530413 25848 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:23.530532 25848 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:23.530581 25848 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:23.575444 25848 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.131:43351
I20240502 21:25:23.575470 25976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.131:43351 every 8 connection(s)
I20240502 21:25:23.576610 25848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 21:25:23.581676 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 25848
I20240502 21:25:23.581851 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 21:25:23.584913 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.132:0
--local_ip_for_outbound_sockets=127.24.226.132
--webserver_interface=127.24.226.132
--webserver_port=0
--tserver_master_addrs=127.24.226.190:36021
--builtin_ntp_servers=127.24.226.148:35721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 21:25:23.589560 25977 heartbeater.cc:347] Connected to a master server at 127.24.226.190:36021
I20240502 21:25:23.589815 25977 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:23.590688 25977 heartbeater.cc:511] Master 127.24.226.190:36021 requested a full tablet report, sending...
I20240502 21:25:23.592010 25525 ts_manager.cc:194] Registered new tserver with Master: 848792f66684491e974a2ff9faae212f (127.24.226.131:43351)
I20240502 21:25:23.593284 25525 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.131:55683
I20240502 21:25:23.784987 25714 heartbeater.cc:503] Master 127.24.226.190:36021 was elected leader, sending a full tablet report...
W20240502 21:25:23.948689 25980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:23.948942 25980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:23.949079 25980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:23.955718 25980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:23.955936 25980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.132
I20240502 21:25:23.964289 25980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:35721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.24.226.132
--webserver_port=0
--tserver_master_addrs=127.24.226.190:36021
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.132
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:23.965348 25980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:23.966951 25980 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:23.979578 25986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:23.979538 25989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:23.979732 25987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:23.980778 25980 server_base.cc:1034] running on GCE node
I20240502 21:25:23.981292 25980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:23.982043 25980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:23.983264 25980 hybrid_clock.cc:648] HybridClock initialized: now 1714685123983177 us; error 38 us; skew 500 ppm
I20240502 21:25:23.983580 25980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:23.985931 25980 webserver.cc:460] Webserver started at http://127.24.226.132:44397/ using document root <none> and password file <none>
I20240502 21:25:23.986618 25980 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:23.986760 25980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:23.987118 25980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:23.989715 25980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "972103a3fe8e4f32a5071ea9f21e9b61"
format_stamp: "Formatted at 2024-05-02 21:25:23 on dist-test-slave-41k1"
I20240502 21:25:23.990396 25980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "972103a3fe8e4f32a5071ea9f21e9b61"
format_stamp: "Formatted at 2024-05-02 21:25:23 on dist-test-slave-41k1"
I20240502 21:25:23.995043 25980 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20240502 21:25:23.998359 25995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:23.999459 25980 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 21:25:23.999593 25980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "972103a3fe8e4f32a5071ea9f21e9b61"
format_stamp: "Formatted at 2024-05-02 21:25:23 on dist-test-slave-41k1"
I20240502 21:25:23.999742 25980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:24.046252 25980 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:24.047315 25980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:24.047621 25980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:24.048772 25980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:24.050679 25980 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:24.050756 25980 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:24.050943 25980 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:24.050992 25980 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:24.093531 25980 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.132:42535
I20240502 21:25:24.093698 26108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.132:42535 every 8 connection(s)
I20240502 21:25:24.094759 25980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 21:25:24.103338 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 25980
I20240502 21:25:24.103482 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 21:25:24.106560 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.133:0
--local_ip_for_outbound_sockets=127.24.226.133
--webserver_interface=127.24.226.133
--webserver_port=0
--tserver_master_addrs=127.24.226.190:36021
--builtin_ntp_servers=127.24.226.148:35721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 21:25:24.108922 26109 heartbeater.cc:347] Connected to a master server at 127.24.226.190:36021
I20240502 21:25:24.109236 26109 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:24.110311 26109 heartbeater.cc:511] Master 127.24.226.190:36021 requested a full tablet report, sending...
I20240502 21:25:24.111840 25525 ts_manager.cc:194] Registered new tserver with Master: 972103a3fe8e4f32a5071ea9f21e9b61 (127.24.226.132:42535)
I20240502 21:25:24.112654 25525 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.132:43235
I20240502 21:25:24.187192 25845 heartbeater.cc:503] Master 127.24.226.190:36021 was elected leader, sending a full tablet report...
W20240502 21:25:24.477273 26113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:24.477550 26113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:24.477706 26113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:24.485174 26113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:24.485354 26113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.133
I20240502 21:25:24.494190 26113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:35721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.24.226.133
--webserver_port=0
--tserver_master_addrs=127.24.226.190:36021
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.133
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:24.495442 26113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:24.497193 26113 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:24.508900 26118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:24.508975 26119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:24.509009 26121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:24.511013 26113 server_base.cc:1034] running on GCE node
I20240502 21:25:24.511525 26113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:24.512374 26113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:24.513630 26113 hybrid_clock.cc:648] HybridClock initialized: now 1714685124513550 us; error 42 us; skew 500 ppm
I20240502 21:25:24.513926 26113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:24.516916 26113 webserver.cc:460] Webserver started at http://127.24.226.133:46135/ using document root <none> and password file <none>
I20240502 21:25:24.517643 26113 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:24.517798 26113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:24.518108 26113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:24.520778 26113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "a44a5d67d0804db7878f0f0894a58925"
format_stamp: "Formatted at 2024-05-02 21:25:24 on dist-test-slave-41k1"
I20240502 21:25:24.521502 26113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "a44a5d67d0804db7878f0f0894a58925"
format_stamp: "Formatted at 2024-05-02 21:25:24 on dist-test-slave-41k1"
I20240502 21:25:24.527753 26113 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20240502 21:25:24.531008 26127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:24.532162 26113 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:24.532302 26113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "a44a5d67d0804db7878f0f0894a58925"
format_stamp: "Formatted at 2024-05-02 21:25:24 on dist-test-slave-41k1"
I20240502 21:25:24.532498 26113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:24.568056 26113 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:24.568842 26113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:24.569171 26113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:24.570055 26113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:24.571453 26113 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:24.571550 26113 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:24.571695 26113 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:24.571755 26113 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:24.595566 25977 heartbeater.cc:503] Master 127.24.226.190:36021 was elected leader, sending a full tablet report...
I20240502 21:25:24.609640 26113 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.133:38681
I20240502 21:25:24.609737 26240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.133:38681 every 8 connection(s)
I20240502 21:25:24.610818 26113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:25:24.614012 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 26113
I20240502 21:25:24.614228 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 21:25:24.624267 26241 heartbeater.cc:347] Connected to a master server at 127.24.226.190:36021
I20240502 21:25:24.624565 26241 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:24.625351 26241 heartbeater.cc:511] Master 127.24.226.190:36021 requested a full tablet report, sending...
I20240502 21:25:24.626507 25525 ts_manager.cc:194] Registered new tserver with Master: a44a5d67d0804db7878f0f0894a58925 (127.24.226.133:38681)
I20240502 21:25:24.627198 25525 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.133:39621
I20240502 21:25:24.630666 25482 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 21:25:24.651356 25525 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39288:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 21:25:24.681618 25648 tablet_service.cc:1467] Processing CreateTablet for tablet 129b14080f1d4586a03b3b59efdb5ca4 (DEFAULT_TABLE table=TestTable [id=1e308005a28649149d246d61b27cfb34]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:24.682869 25648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 129b14080f1d4586a03b3b59efdb5ca4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:24.683092 26043 tablet_service.cc:1467] Processing CreateTablet for tablet 129b14080f1d4586a03b3b59efdb5ca4 (DEFAULT_TABLE table=TestTable [id=1e308005a28649149d246d61b27cfb34]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:24.684324 26043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 129b14080f1d4586a03b3b59efdb5ca4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:24.686403 25912 tablet_service.cc:1467] Processing CreateTablet for tablet 129b14080f1d4586a03b3b59efdb5ca4 (DEFAULT_TABLE table=TestTable [id=1e308005a28649149d246d61b27cfb34]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:24.687731 25912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 129b14080f1d4586a03b3b59efdb5ca4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:24.694835 26260 tablet_bootstrap.cc:492] T 129b14080f1d4586a03b3b59efdb5ca4 P 972103a3fe8e4f32a5071ea9f21e9b61: Bootstrap starting.
I20240502 21:25:24.697011 26261 tablet_bootstrap.cc:492] T 129b14080f1d4586a03b3b59efdb5ca4 P 3bc83d3565d14e8399843f8b0f593e53: Bootstrap starting.
I20240502 21:25:24.700134 26260 tablet_bootstrap.cc:654] T 129b14080f1d4586a03b3b59efdb5ca4 P 972103a3fe8e4f32a5071ea9f21e9b61: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:24.701076 26261 tablet_bootstrap.cc:654] T 129b14080f1d4586a03b3b59efdb5ca4 P 3bc83d3565d14e8399843f8b0f593e53: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:24.701092 26262 tablet_bootstrap.cc:492] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f: Bootstrap starting.
I20240502 21:25:24.701802 26260 log.cc:824] T 129b14080f1d4586a03b3b59efdb5ca4 P 972103a3fe8e4f32a5071ea9f21e9b61: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:24.702848 26261 log.cc:824] T 129b14080f1d4586a03b3b59efdb5ca4 P 3bc83d3565d14e8399843f8b0f593e53: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:24.704694 26262 tablet_bootstrap.cc:654] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:24.704929 26260 tablet_bootstrap.cc:492] T 129b14080f1d4586a03b3b59efdb5ca4 P 972103a3fe8e4f32a5071ea9f21e9b61: No bootstrap required, opened a new log
I20240502 21:25:24.705264 26260 ts_tablet_manager.cc:1367] T 129b14080f1d4586a03b3b59efdb5ca4 P 972103a3fe8e4f32a5071ea9f21e9b61: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20240502 21:25:24.706059 26262 log.cc:824] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:24.706313 26261 tablet_bootstrap.cc:492] T 129b14080f1d4586a03b3b59efdb5ca4 P 3bc83d3565d14e8399843f8b0f593e53: No bootstrap required, opened a new log
I20240502 21:25:24.706588 26261 ts_tablet_manager.cc:1367] T 129b14080f1d4586a03b3b59efdb5ca4 P 3bc83d3565d14e8399843f8b0f593e53: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20240502 21:25:24.708680 26262 tablet_bootstrap.cc:492] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f: No bootstrap required, opened a new log
I20240502 21:25:24.708940 26262 ts_tablet_manager.cc:1367] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20240502 21:25:24.715582 26261 raft_consensus.cc:348] T 129b14080f1d4586a03b3b59efdb5ca4 P 3bc83d3565d14e8399843f8b0f593e53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc83d3565d14e8399843f8b0f593e53" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39653 } } peers { permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 } } peers { permanent_uuid: "848792f66684491e974a2ff9faae212f" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 43351 } }
I20240502 21:25:24.715569 26260 raft_consensus.cc:348] T 129b14080f1d4586a03b3b59efdb5ca4 P 972103a3fe8e4f32a5071ea9f21e9b61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc83d3565d14e8399843f8b0f593e53" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39653 } } peers { permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 } } peers { permanent_uuid: "848792f66684491e974a2ff9faae212f" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 43351 } }
I20240502 21:25:24.715978 26261 raft_consensus.cc:374] T 129b14080f1d4586a03b3b59efdb5ca4 P 3bc83d3565d14e8399843f8b0f593e53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:24.716079 26260 raft_consensus.cc:374] T 129b14080f1d4586a03b3b59efdb5ca4 P 972103a3fe8e4f32a5071ea9f21e9b61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:24.716128 26261 raft_consensus.cc:729] T 129b14080f1d4586a03b3b59efdb5ca4 P 3bc83d3565d14e8399843f8b0f593e53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bc83d3565d14e8399843f8b0f593e53, State: Initialized, Role: FOLLOWER
I20240502 21:25:24.715898 26262 raft_consensus.cc:348] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc83d3565d14e8399843f8b0f593e53" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39653 } } peers { permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 } } peers { permanent_uuid: "848792f66684491e974a2ff9faae212f" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 43351 } }
I20240502 21:25:24.716329 26260 raft_consensus.cc:729] T 129b14080f1d4586a03b3b59efdb5ca4 P 972103a3fe8e4f32a5071ea9f21e9b61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 972103a3fe8e4f32a5071ea9f21e9b61, State: Initialized, Role: FOLLOWER
I20240502 21:25:24.716516 26262 raft_consensus.cc:374] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:24.716674 26262 raft_consensus.cc:729] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 848792f66684491e974a2ff9faae212f, State: Initialized, Role: FOLLOWER
I20240502 21:25:24.716925 26261 consensus_queue.cc:260] T 129b14080f1d4586a03b3b59efdb5ca4 P 3bc83d3565d14e8399843f8b0f593e53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc83d3565d14e8399843f8b0f593e53" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39653 } } peers { permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 } } peers { permanent_uuid: "848792f66684491e974a2ff9faae212f" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 43351 } }
I20240502 21:25:24.718449 26260 consensus_queue.cc:260] T 129b14080f1d4586a03b3b59efdb5ca4 P 972103a3fe8e4f32a5071ea9f21e9b61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc83d3565d14e8399843f8b0f593e53" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39653 } } peers { permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 } } peers { permanent_uuid: "848792f66684491e974a2ff9faae212f" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 43351 } }
I20240502 21:25:24.718948 26261 ts_tablet_manager.cc:1398] T 129b14080f1d4586a03b3b59efdb5ca4 P 3bc83d3565d14e8399843f8b0f593e53: Time spent starting tablet: real 0.012s user 0.006s sys 0.006s
I20240502 21:25:24.720088 26262 consensus_queue.cc:260] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc83d3565d14e8399843f8b0f593e53" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39653 } } peers { permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 } } peers { permanent_uuid: "848792f66684491e974a2ff9faae212f" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 43351 } }
I20240502 21:25:24.720713 26266 raft_consensus.cc:482] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:24.720784 26109 heartbeater.cc:503] Master 127.24.226.190:36021 was elected leader, sending a full tablet report...
I20240502 21:25:24.720922 26266 raft_consensus.cc:504] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc83d3565d14e8399843f8b0f593e53" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39653 } } peers { permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 } } peers { permanent_uuid: "848792f66684491e974a2ff9faae212f" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 43351 } }
I20240502 21:25:24.724627 26260 ts_tablet_manager.cc:1398] T 129b14080f1d4586a03b3b59efdb5ca4 P 972103a3fe8e4f32a5071ea9f21e9b61: Time spent starting tablet: real 0.019s user 0.009s sys 0.005s
I20240502 21:25:24.727867 26266 leader_election.cc:288] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3bc83d3565d14e8399843f8b0f593e53 (127.24.226.129:39653), 972103a3fe8e4f32a5071ea9f21e9b61 (127.24.226.132:42535)
I20240502 21:25:24.728152 26262 ts_tablet_manager.cc:1398] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f: Time spent starting tablet: real 0.019s user 0.010s sys 0.009s
I20240502 21:25:24.736414 25668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "129b14080f1d4586a03b3b59efdb5ca4" candidate_uuid: "848792f66684491e974a2ff9faae212f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bc83d3565d14e8399843f8b0f593e53" is_pre_election: true
I20240502 21:25:24.736855 25668 raft_consensus.cc:2454] T 129b14080f1d4586a03b3b59efdb5ca4 P 3bc83d3565d14e8399843f8b0f593e53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 848792f66684491e974a2ff9faae212f in term 0.
I20240502 21:25:24.737552 25865 leader_election.cc:302] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bc83d3565d14e8399843f8b0f593e53, 848792f66684491e974a2ff9faae212f; no voters:
I20240502 21:25:24.737874 26266 raft_consensus.cc:2781] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:24.737977 26266 raft_consensus.cc:482] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:24.738072 26266 raft_consensus.cc:3037] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:24.738023 26064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "129b14080f1d4586a03b3b59efdb5ca4" candidate_uuid: "848792f66684491e974a2ff9faae212f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" is_pre_election: true
I20240502 21:25:24.738628 26064 raft_consensus.cc:2454] T 129b14080f1d4586a03b3b59efdb5ca4 P 972103a3fe8e4f32a5071ea9f21e9b61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 848792f66684491e974a2ff9faae212f in term 0.
W20240502 21:25:24.770051 25715 tablet.cc:2369] T 129b14080f1d4586a03b3b59efdb5ca4 P 3bc83d3565d14e8399843f8b0f593e53: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:25:24.796648 26266 raft_consensus.cc:504] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc83d3565d14e8399843f8b0f593e53" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39653 } } peers { permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 } } peers { permanent_uuid: "848792f66684491e974a2ff9faae212f" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 43351 } }
I20240502 21:25:24.797609 26266 leader_election.cc:288] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [CANDIDATE]: Term 1 election: Requested vote from peers 3bc83d3565d14e8399843f8b0f593e53 (127.24.226.129:39653), 972103a3fe8e4f32a5071ea9f21e9b61 (127.24.226.132:42535)
I20240502 21:25:24.797976 25668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "129b14080f1d4586a03b3b59efdb5ca4" candidate_uuid: "848792f66684491e974a2ff9faae212f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bc83d3565d14e8399843f8b0f593e53"
I20240502 21:25:24.798039 26064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "129b14080f1d4586a03b3b59efdb5ca4" candidate_uuid: "848792f66684491e974a2ff9faae212f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "972103a3fe8e4f32a5071ea9f21e9b61"
I20240502 21:25:24.798184 25668 raft_consensus.cc:3037] T 129b14080f1d4586a03b3b59efdb5ca4 P 3bc83d3565d14e8399843f8b0f593e53 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:24.798267 26064 raft_consensus.cc:3037] T 129b14080f1d4586a03b3b59efdb5ca4 P 972103a3fe8e4f32a5071ea9f21e9b61 [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:25:24.829425 25978 tablet.cc:2369] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:25:24.836666 25668 raft_consensus.cc:2454] T 129b14080f1d4586a03b3b59efdb5ca4 P 3bc83d3565d14e8399843f8b0f593e53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 848792f66684491e974a2ff9faae212f in term 1.
I20240502 21:25:24.837049 26064 raft_consensus.cc:2454] T 129b14080f1d4586a03b3b59efdb5ca4 P 972103a3fe8e4f32a5071ea9f21e9b61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 848792f66684491e974a2ff9faae212f in term 1.
I20240502 21:25:24.837180 25865 leader_election.cc:302] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bc83d3565d14e8399843f8b0f593e53, 848792f66684491e974a2ff9faae212f; no voters:
I20240502 21:25:24.837608 26266 raft_consensus.cc:2781] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:24.837769 26266 raft_consensus.cc:686] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [term 1 LEADER]: Becoming Leader. State: Replica: 848792f66684491e974a2ff9faae212f, State: Running, Role: LEADER
I20240502 21:25:24.838353 26266 consensus_queue.cc:237] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc83d3565d14e8399843f8b0f593e53" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39653 } } peers { permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 } } peers { permanent_uuid: "848792f66684491e974a2ff9faae212f" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 43351 } }
I20240502 21:25:24.843048 25524 catalog_manager.cc:5484] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f reported cstate change: term changed from 0 to 1, leader changed from <none> to 848792f66684491e974a2ff9faae212f (127.24.226.131). New cstate: current_term: 1 leader_uuid: "848792f66684491e974a2ff9faae212f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bc83d3565d14e8399843f8b0f593e53" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "848792f66684491e974a2ff9faae212f" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 43351 } health_report { overall_health: HEALTHY } } }
W20240502 21:25:24.846738 26110 tablet.cc:2369] T 129b14080f1d4586a03b3b59efdb5ca4 P 972103a3fe8e4f32a5071ea9f21e9b61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:25:24.863571 25482 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 21:25:24.866045 25482 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 3bc83d3565d14e8399843f8b0f593e53 to finish bootstrapping
I20240502 21:25:24.875053 25482 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4f5fa89cbeac47e5bf611a21252c7732 to finish bootstrapping
I20240502 21:25:24.883911 25482 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 848792f66684491e974a2ff9faae212f to finish bootstrapping
I20240502 21:25:24.891595 25482 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 972103a3fe8e4f32a5071ea9f21e9b61 to finish bootstrapping
I20240502 21:25:24.898672 25482 ts_itest-base.cc:233] Waiting for 0 tablets on tserver a44a5d67d0804db7878f0f0894a58925 to finish bootstrapping
I20240502 21:25:25.325131 26266 consensus_queue.cc:1035] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bc83d3565d14e8399843f8b0f593e53" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:25.345058 26266 consensus_queue.cc:1035] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [LEADER]: Connected to new peer: Peer: permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:25.409111 25648 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:25.419836 25912 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:25.443475 25780 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:25.462179 26043 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:25.474233 26175 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:25.629626 26241 heartbeater.cc:503] Master 127.24.226.190:36021 was elected leader, sending a full tablet report...
I20240502 21:25:26.057072 25648 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:26.057214 26043 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:26.059334 25912 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:26.064069 26175 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:26.072938 25780 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:26.234483 25912 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:26.238094 25780 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:26.240419 26043 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:26.240512 26175 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:26.253049 25648 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:26.420493 25648 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:26.423513 25780 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:26.435786 25912 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:26.436852 26043 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:26.446334 26175 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:26.948135 25524 ts_manager.cc:292] Set tserver state for 3bc83d3565d14e8399843f8b0f593e53 to MAINTENANCE_MODE
I20240502 21:25:27.472081 25912 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:27.473800 25648 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:27.475272 25780 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:27.485987 26175 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:27.496310 26043 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:27.644145 25648 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:27.649776 25912 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:27.651463 25780 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:27.658819 26043 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:27.668954 26175 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:27.829030 25780 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:27.835008 25648 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:27.842056 25912 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:27.844795 26175 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:27.857954 26043 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:27.921463 25932 consensus_queue.cc:237] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3bc83d3565d14e8399843f8b0f593e53" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39653 } attrs { replace: true } } peers { permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 } } peers { permanent_uuid: "848792f66684491e974a2ff9faae212f" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 43351 } }
I20240502 21:25:27.923527 26064 raft_consensus.cc:1261] T 129b14080f1d4586a03b3b59efdb5ca4 P 972103a3fe8e4f32a5071ea9f21e9b61 [term 1 FOLLOWER]: Refusing update from remote peer 848792f66684491e974a2ff9faae212f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:25:27.923884 25668 raft_consensus.cc:1261] T 129b14080f1d4586a03b3b59efdb5ca4 P 3bc83d3565d14e8399843f8b0f593e53 [term 1 FOLLOWER]: Refusing update from remote peer 848792f66684491e974a2ff9faae212f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:25:27.924034 26553 consensus_queue.cc:1035] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [LEADER]: Connected to new peer: Peer: permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:25:27.924512 26323 consensus_queue.cc:1035] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bc83d3565d14e8399843f8b0f593e53" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39653 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:25:27.927088 26553 raft_consensus.cc:2932] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3bc83d3565d14e8399843f8b0f593e53" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39653 } attrs { replace: true } } peers { permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 } } peers { permanent_uuid: "848792f66684491e974a2ff9faae212f" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 43351 } } }
I20240502 21:25:27.927908 26064 raft_consensus.cc:2932] T 129b14080f1d4586a03b3b59efdb5ca4 P 972103a3fe8e4f32a5071ea9f21e9b61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3bc83d3565d14e8399843f8b0f593e53" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39653 } attrs { replace: true } } peers { permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 } } peers { permanent_uuid: "848792f66684491e974a2ff9faae212f" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 43351 } } }
I20240502 21:25:27.928228 25668 raft_consensus.cc:2932] T 129b14080f1d4586a03b3b59efdb5ca4 P 3bc83d3565d14e8399843f8b0f593e53 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3bc83d3565d14e8399843f8b0f593e53" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39653 } attrs { replace: true } } peers { permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 } } peers { permanent_uuid: "848792f66684491e974a2ff9faae212f" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 43351 } } }
I20240502 21:25:27.995792 25525 catalog_manager.cc:5484] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f reported cstate change: config changed from index -1 to 2. New cstate: current_term: 1 leader_uuid: "848792f66684491e974a2ff9faae212f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3bc83d3565d14e8399843f8b0f593e53" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39653 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "848792f66684491e974a2ff9faae212f" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 43351 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:28.008530 25932 consensus_queue.cc:237] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3bc83d3565d14e8399843f8b0f593e53" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39653 } attrs { replace: true } } peers { permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 } } peers { permanent_uuid: "848792f66684491e974a2ff9faae212f" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 43351 } } peers { permanent_uuid: "a44a5d67d0804db7878f0f0894a58925" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 38681 } attrs { promote: true } }
I20240502 21:25:28.011811 25668 raft_consensus.cc:1261] T 129b14080f1d4586a03b3b59efdb5ca4 P 3bc83d3565d14e8399843f8b0f593e53 [term 1 FOLLOWER]: Refusing update from remote peer 848792f66684491e974a2ff9faae212f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:25:28.011833 26064 raft_consensus.cc:1261] T 129b14080f1d4586a03b3b59efdb5ca4 P 972103a3fe8e4f32a5071ea9f21e9b61 [term 1 FOLLOWER]: Refusing update from remote peer 848792f66684491e974a2ff9faae212f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:25:28.013064 26323 consensus_queue.cc:1035] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [LEADER]: Connected to new peer: Peer: permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:25:28.013566 26553 consensus_queue.cc:1035] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bc83d3565d14e8399843f8b0f593e53" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39653 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:25:28.016177 26553 raft_consensus.cc:2932] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER a44a5d67d0804db7878f0f0894a58925 (127.24.226.133) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3bc83d3565d14e8399843f8b0f593e53" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39653 } attrs { replace: true } } peers { permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 } } peers { permanent_uuid: "848792f66684491e974a2ff9faae212f" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 43351 } } peers { permanent_uuid: "a44a5d67d0804db7878f0f0894a58925" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 38681 } attrs { promote: true } } }
I20240502 21:25:28.017349 26064 raft_consensus.cc:2932] T 129b14080f1d4586a03b3b59efdb5ca4 P 972103a3fe8e4f32a5071ea9f21e9b61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER a44a5d67d0804db7878f0f0894a58925 (127.24.226.133) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3bc83d3565d14e8399843f8b0f593e53" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39653 } attrs { replace: true } } peers { permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 } } peers { permanent_uuid: "848792f66684491e974a2ff9faae212f" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 43351 } } peers { permanent_uuid: "a44a5d67d0804db7878f0f0894a58925" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 38681 } attrs { promote: true } } }
I20240502 21:25:28.020265 25668 raft_consensus.cc:2932] T 129b14080f1d4586a03b3b59efdb5ca4 P 3bc83d3565d14e8399843f8b0f593e53 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER a44a5d67d0804db7878f0f0894a58925 (127.24.226.133) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3bc83d3565d14e8399843f8b0f593e53" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39653 } attrs { replace: true } } peers { permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 } } peers { permanent_uuid: "848792f66684491e974a2ff9faae212f" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 43351 } } peers { permanent_uuid: "a44a5d67d0804db7878f0f0894a58925" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 38681 } attrs { promote: true } } }
W20240502 21:25:28.024982 25866 consensus_peers.cc:485] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f -> Peer a44a5d67d0804db7878f0f0894a58925 (127.24.226.133:38681): Couldn't send request to peer a44a5d67d0804db7878f0f0894a58925. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 129b14080f1d4586a03b3b59efdb5ca4. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:28.103468 25512 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 129b14080f1d4586a03b3b59efdb5ca4 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:25:28.104944 25524 catalog_manager.cc:5484] T 129b14080f1d4586a03b3b59efdb5ca4 P 3bc83d3565d14e8399843f8b0f593e53 reported cstate change: config changed from index 2 to 3, NON_VOTER a44a5d67d0804db7878f0f0894a58925 (127.24.226.133) added. New cstate: current_term: 1 leader_uuid: "848792f66684491e974a2ff9faae212f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3bc83d3565d14e8399843f8b0f593e53" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39653 } attrs { replace: true } } peers { permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 } } peers { permanent_uuid: "848792f66684491e974a2ff9faae212f" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 43351 } } peers { permanent_uuid: "a44a5d67d0804db7878f0f0894a58925" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 38681 } attrs { promote: true } } }
I20240502 21:25:28.618085 26579 ts_tablet_manager.cc:902] T 129b14080f1d4586a03b3b59efdb5ca4 P a44a5d67d0804db7878f0f0894a58925: Initiating tablet copy from peer 848792f66684491e974a2ff9faae212f (127.24.226.131:43351)
I20240502 21:25:28.619977 26579 tablet_copy_client.cc:308] T 129b14080f1d4586a03b3b59efdb5ca4 P a44a5d67d0804db7878f0f0894a58925: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.131:43351
I20240502 21:25:28.626503 25952 tablet_copy_service.cc:140] P 848792f66684491e974a2ff9faae212f: Received BeginTabletCopySession request for tablet 129b14080f1d4586a03b3b59efdb5ca4 from peer a44a5d67d0804db7878f0f0894a58925 ({username='slave'} at 127.24.226.133:51539)
I20240502 21:25:28.626696 25952 tablet_copy_service.cc:161] P 848792f66684491e974a2ff9faae212f: Beginning new tablet copy session on tablet 129b14080f1d4586a03b3b59efdb5ca4 from peer a44a5d67d0804db7878f0f0894a58925 at {username='slave'} at 127.24.226.133:51539: session id = a44a5d67d0804db7878f0f0894a58925-129b14080f1d4586a03b3b59efdb5ca4
I20240502 21:25:28.629354 25952 tablet_copy_source_session.cc:205] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:25:28.633818 26579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 129b14080f1d4586a03b3b59efdb5ca4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:28.646014 26579 tablet_copy_client.cc:783] T 129b14080f1d4586a03b3b59efdb5ca4 P a44a5d67d0804db7878f0f0894a58925: tablet copy: Starting download of 0 data blocks...
I20240502 21:25:28.646380 26579 tablet_copy_client.cc:647] T 129b14080f1d4586a03b3b59efdb5ca4 P a44a5d67d0804db7878f0f0894a58925: tablet copy: Starting download of 1 WAL segments...
I20240502 21:25:28.648464 26579 tablet_copy_client.cc:523] T 129b14080f1d4586a03b3b59efdb5ca4 P a44a5d67d0804db7878f0f0894a58925: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:25:28.719930 26579 tablet_bootstrap.cc:492] T 129b14080f1d4586a03b3b59efdb5ca4 P a44a5d67d0804db7878f0f0894a58925: Bootstrap starting.
I20240502 21:25:28.729020 26579 log.cc:824] T 129b14080f1d4586a03b3b59efdb5ca4 P a44a5d67d0804db7878f0f0894a58925: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:28.732254 26579 tablet_bootstrap.cc:492] T 129b14080f1d4586a03b3b59efdb5ca4 P a44a5d67d0804db7878f0f0894a58925: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:25:28.732625 26579 tablet_bootstrap.cc:492] T 129b14080f1d4586a03b3b59efdb5ca4 P a44a5d67d0804db7878f0f0894a58925: Bootstrap complete.
I20240502 21:25:28.732933 26579 ts_tablet_manager.cc:1367] T 129b14080f1d4586a03b3b59efdb5ca4 P a44a5d67d0804db7878f0f0894a58925: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.010s
I20240502 21:25:28.738688 26579 raft_consensus.cc:348] T 129b14080f1d4586a03b3b59efdb5ca4 P a44a5d67d0804db7878f0f0894a58925 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3bc83d3565d14e8399843f8b0f593e53" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39653 } attrs { replace: true } } peers { permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 } } peers { permanent_uuid: "848792f66684491e974a2ff9faae212f" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 43351 } } peers { permanent_uuid: "a44a5d67d0804db7878f0f0894a58925" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 38681 } attrs { promote: true } }
I20240502 21:25:28.739044 26579 raft_consensus.cc:729] T 129b14080f1d4586a03b3b59efdb5ca4 P a44a5d67d0804db7878f0f0894a58925 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a44a5d67d0804db7878f0f0894a58925, State: Initialized, Role: LEARNER
I20240502 21:25:28.739432 26579 consensus_queue.cc:260] T 129b14080f1d4586a03b3b59efdb5ca4 P a44a5d67d0804db7878f0f0894a58925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3bc83d3565d14e8399843f8b0f593e53" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39653 } attrs { replace: true } } peers { permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 } } peers { permanent_uuid: "848792f66684491e974a2ff9faae212f" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 43351 } } peers { permanent_uuid: "a44a5d67d0804db7878f0f0894a58925" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 38681 } attrs { promote: true } }
I20240502 21:25:28.741010 26579 ts_tablet_manager.cc:1398] T 129b14080f1d4586a03b3b59efdb5ca4 P a44a5d67d0804db7878f0f0894a58925: Time spent starting tablet: real 0.008s user 0.006s sys 0.002s
I20240502 21:25:28.742182 25952 tablet_copy_service.cc:342] P 848792f66684491e974a2ff9faae212f: Request end of tablet copy session a44a5d67d0804db7878f0f0894a58925-129b14080f1d4586a03b3b59efdb5ca4 received from {username='slave'} at 127.24.226.133:51539
I20240502 21:25:28.742352 25952 tablet_copy_service.cc:433] P 848792f66684491e974a2ff9faae212f: ending tablet copy session a44a5d67d0804db7878f0f0894a58925-129b14080f1d4586a03b3b59efdb5ca4 on tablet 129b14080f1d4586a03b3b59efdb5ca4 with peer a44a5d67d0804db7878f0f0894a58925
I20240502 21:25:29.144646 26196 raft_consensus.cc:1203] T 129b14080f1d4586a03b3b59efdb5ca4 P a44a5d67d0804db7878f0f0894a58925 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20240502 21:25:29.709686 26603 raft_consensus.cc:1050] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f: attempting to promote NON_VOTER a44a5d67d0804db7878f0f0894a58925 to VOTER
I20240502 21:25:29.710404 26603 consensus_queue.cc:237] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3bc83d3565d14e8399843f8b0f593e53" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39653 } attrs { replace: true } } peers { permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 } } peers { permanent_uuid: "848792f66684491e974a2ff9faae212f" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 43351 } } peers { permanent_uuid: "a44a5d67d0804db7878f0f0894a58925" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38681 } attrs { promote: false } }
I20240502 21:25:29.712731 26196 raft_consensus.cc:1261] T 129b14080f1d4586a03b3b59efdb5ca4 P a44a5d67d0804db7878f0f0894a58925 [term 1 LEARNER]: Refusing update from remote peer 848792f66684491e974a2ff9faae212f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:25:29.713090 26064 raft_consensus.cc:1261] T 129b14080f1d4586a03b3b59efdb5ca4 P 972103a3fe8e4f32a5071ea9f21e9b61 [term 1 FOLLOWER]: Refusing update from remote peer 848792f66684491e974a2ff9faae212f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:25:29.713477 25668 raft_consensus.cc:1261] T 129b14080f1d4586a03b3b59efdb5ca4 P 3bc83d3565d14e8399843f8b0f593e53 [term 1 FOLLOWER]: Refusing update from remote peer 848792f66684491e974a2ff9faae212f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:25:29.713420 26603 consensus_queue.cc:1035] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44a5d67d0804db7878f0f0894a58925" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38681 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:25:29.713690 26621 consensus_queue.cc:1035] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [LEADER]: Connected to new peer: Peer: permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:25:29.714012 26619 consensus_queue.cc:1035] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bc83d3565d14e8399843f8b0f593e53" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39653 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:25:29.721629 26603 raft_consensus.cc:2932] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, a44a5d67d0804db7878f0f0894a58925 (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3bc83d3565d14e8399843f8b0f593e53" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39653 } attrs { replace: true } } peers { permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 } } peers { permanent_uuid: "848792f66684491e974a2ff9faae212f" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 43351 } } peers { permanent_uuid: "a44a5d67d0804db7878f0f0894a58925" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38681 } attrs { promote: false } } }
I20240502 21:25:29.722298 25668 raft_consensus.cc:2932] T 129b14080f1d4586a03b3b59efdb5ca4 P 3bc83d3565d14e8399843f8b0f593e53 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, a44a5d67d0804db7878f0f0894a58925 (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3bc83d3565d14e8399843f8b0f593e53" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39653 } attrs { replace: true } } peers { permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 } } peers { permanent_uuid: "848792f66684491e974a2ff9faae212f" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 43351 } } peers { permanent_uuid: "a44a5d67d0804db7878f0f0894a58925" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38681 } attrs { promote: false } } }
I20240502 21:25:29.722298 26064 raft_consensus.cc:2932] T 129b14080f1d4586a03b3b59efdb5ca4 P 972103a3fe8e4f32a5071ea9f21e9b61 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, a44a5d67d0804db7878f0f0894a58925 (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3bc83d3565d14e8399843f8b0f593e53" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39653 } attrs { replace: true } } peers { permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 } } peers { permanent_uuid: "848792f66684491e974a2ff9faae212f" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 43351 } } peers { permanent_uuid: "a44a5d67d0804db7878f0f0894a58925" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38681 } attrs { promote: false } } }
I20240502 21:25:29.725464 26196 raft_consensus.cc:2932] T 129b14080f1d4586a03b3b59efdb5ca4 P a44a5d67d0804db7878f0f0894a58925 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, a44a5d67d0804db7878f0f0894a58925 (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3bc83d3565d14e8399843f8b0f593e53" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39653 } attrs { replace: true } } peers { permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 } } peers { permanent_uuid: "848792f66684491e974a2ff9faae212f" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 43351 } } peers { permanent_uuid: "a44a5d67d0804db7878f0f0894a58925" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38681 } attrs { promote: false } } }
I20240502 21:25:29.725914 25525 catalog_manager.cc:5484] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f reported cstate change: config changed from index 3 to 4, a44a5d67d0804db7878f0f0894a58925 (127.24.226.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "848792f66684491e974a2ff9faae212f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3bc83d3565d14e8399843f8b0f593e53" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39653 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "848792f66684491e974a2ff9faae212f" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 43351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44a5d67d0804db7878f0f0894a58925" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38681 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:29.737325 25932 consensus_queue.cc:237] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 } } peers { permanent_uuid: "848792f66684491e974a2ff9faae212f" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 43351 } } peers { permanent_uuid: "a44a5d67d0804db7878f0f0894a58925" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38681 } attrs { promote: false } }
I20240502 21:25:29.738739 26196 raft_consensus.cc:1261] T 129b14080f1d4586a03b3b59efdb5ca4 P a44a5d67d0804db7878f0f0894a58925 [term 1 FOLLOWER]: Refusing update from remote peer 848792f66684491e974a2ff9faae212f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:25:29.738787 26064 raft_consensus.cc:1261] T 129b14080f1d4586a03b3b59efdb5ca4 P 972103a3fe8e4f32a5071ea9f21e9b61 [term 1 FOLLOWER]: Refusing update from remote peer 848792f66684491e974a2ff9faae212f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:25:29.739224 26603 consensus_queue.cc:1035] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44a5d67d0804db7878f0f0894a58925" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38681 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:25:29.739464 26619 consensus_queue.cc:1035] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [LEADER]: Connected to new peer: Peer: permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:25:29.741398 26603 raft_consensus.cc:2932] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 3bc83d3565d14e8399843f8b0f593e53 (127.24.226.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 } } peers { permanent_uuid: "848792f66684491e974a2ff9faae212f" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 43351 } } peers { permanent_uuid: "a44a5d67d0804db7878f0f0894a58925" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38681 } attrs { promote: false } } }
I20240502 21:25:29.741856 26064 raft_consensus.cc:2932] T 129b14080f1d4586a03b3b59efdb5ca4 P 972103a3fe8e4f32a5071ea9f21e9b61 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 3bc83d3565d14e8399843f8b0f593e53 (127.24.226.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 } } peers { permanent_uuid: "848792f66684491e974a2ff9faae212f" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 43351 } } peers { permanent_uuid: "a44a5d67d0804db7878f0f0894a58925" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38681 } attrs { promote: false } } }
I20240502 21:25:29.742230 26196 raft_consensus.cc:2932] T 129b14080f1d4586a03b3b59efdb5ca4 P a44a5d67d0804db7878f0f0894a58925 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 3bc83d3565d14e8399843f8b0f593e53 (127.24.226.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 } } peers { permanent_uuid: "848792f66684491e974a2ff9faae212f" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 43351 } } peers { permanent_uuid: "a44a5d67d0804db7878f0f0894a58925" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38681 } attrs { promote: false } } }
I20240502 21:25:29.744944 25522 catalog_manager.cc:5484] T 129b14080f1d4586a03b3b59efdb5ca4 P 848792f66684491e974a2ff9faae212f reported cstate change: config changed from index 4 to 5, VOTER 3bc83d3565d14e8399843f8b0f593e53 (127.24.226.129) evicted. New cstate: current_term: 1 leader_uuid: "848792f66684491e974a2ff9faae212f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "972103a3fe8e4f32a5071ea9f21e9b61" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "848792f66684491e974a2ff9faae212f" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 43351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44a5d67d0804db7878f0f0894a58925" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38681 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:29.745335 25512 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 129b14080f1d4586a03b3b59efdb5ca4 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:25:29.748538 25648 tablet_service.cc:1514] Processing DeleteTablet for tablet 129b14080f1d4586a03b3b59efdb5ca4 with delete_type TABLET_DATA_TOMBSTONED (TS 3bc83d3565d14e8399843f8b0f593e53 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52466
I20240502 21:25:29.749545 26633 tablet_replica.cc:330] T 129b14080f1d4586a03b3b59efdb5ca4 P 3bc83d3565d14e8399843f8b0f593e53: stopping tablet replica
I20240502 21:25:29.749944 26633 raft_consensus.cc:2229] T 129b14080f1d4586a03b3b59efdb5ca4 P 3bc83d3565d14e8399843f8b0f593e53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:25:29.750314 26633 raft_consensus.cc:2258] T 129b14080f1d4586a03b3b59efdb5ca4 P 3bc83d3565d14e8399843f8b0f593e53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:29.751477 26633 ts_tablet_manager.cc:1875] T 129b14080f1d4586a03b3b59efdb5ca4 P 3bc83d3565d14e8399843f8b0f593e53: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:25:29.755834 26633 ts_tablet_manager.cc:1888] T 129b14080f1d4586a03b3b59efdb5ca4 P 3bc83d3565d14e8399843f8b0f593e53: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 21:25:29.755985 26633 log.cc:1196] T 129b14080f1d4586a03b3b59efdb5ca4 P 3bc83d3565d14e8399843f8b0f593e53: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/129b14080f1d4586a03b3b59efdb5ca4
I20240502 21:25:29.756705 25511 catalog_manager.cc:4830] TS 3bc83d3565d14e8399843f8b0f593e53 (127.24.226.129:39653): tablet 129b14080f1d4586a03b3b59efdb5ca4 (table TestTable [id=1e308005a28649149d246d61b27cfb34]) successfully deleted
I20240502 21:25:30.194741 25648 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:30.217361 25780 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:30.218366 26043 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:30.238286 25912 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:30.249104 26175 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:30.372474 25648 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:30.379540 25780 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:30.389122 25912 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:30.390235 26043 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:30.401971 26175 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:30.513201 25648 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:30.513999 25780 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:30.530228 26043 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:30.532385 26175 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:30.532953 25912 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:30.686844 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 25718
I20240502 21:25:30.688117 25839 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:30.786768 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 25718
I20240502 21:25:31.215591 25648 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:31.226033 25912 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:31.226979 26043 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:31.237840 26175 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:31.344029 25648 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:31.352744 25912 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:31.354581 26043 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:31.354630 26175 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:31.508154 25648 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:31.510888 25912 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:31.513018 26175 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:31.519366 26043 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:31.634207 25912 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:31.635529 25648 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:31.649511 26043 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:31.651527 26175 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:31.792688 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 25585
I20240502 21:25:31.793853 25709 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:31.900846 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 25585
I20240502 21:25:31.921368 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 25848
I20240502 21:25:31.922223 25971 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:32.028501 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 25848
I20240502 21:25:32.049218 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 25980
I20240502 21:25:32.050235 26104 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:32.153316 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 25980
I20240502 21:25:32.173679 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 26113
I20240502 21:25:32.174633 26236 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:32.283174 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 26113
I20240502 21:25:32.302227 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 25494
I20240502 21:25:32.303239 25555 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:32.426978 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 25494
2024-05-02T21:25:32Z chronyd exiting
[ OK ] RebalanceIgnoredTserversTest.Basic (10725 ms)
[----------] 1 test from RebalanceIgnoredTserversTest (10725 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/1
I20240502 21:25:32.471343 25482 test_util.cc:268] Using random seed: 1043921937
I20240502 21:25:32.472813 25482 ts_itest-base.cc:112] Starting cluster with:
I20240502 21:25:32.472892 25482 ts_itest-base.cc:113] --------------
I20240502 21:25:32.472946 25482 ts_itest-base.cc:114] 5 tablet servers
I20240502 21:25:32.473027 25482 ts_itest-base.cc:115] 3 replicas per TS
I20240502 21:25:32.473070 25482 ts_itest-base.cc:116] --------------
2024-05-02T21:25:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T21:25:32Z Disabled control of system clock
I20240502 21:25:32.492882 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.226.190:43225
--webserver_interface=127.24.226.190
--webserver_port=0
--builtin_ntp_servers=127.24.226.148:41147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.226.190:43225
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 21:25:32.783617 26864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:32.783849 26864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:32.783953 26864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:32.790218 26864 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:32.790305 26864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 21:25:32.790339 26864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:32.790387 26864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 21:25:32.790442 26864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 21:25:32.797938 26864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:41147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.24.226.190:43225
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.226.190:43225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.24.226.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:32.798977 26864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:32.800521 26864 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:32.809792 26869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:32.809767 26872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:32.809818 26870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:32.810477 26864 server_base.cc:1034] running on GCE node
I20240502 21:25:32.811053 26864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:32.812111 26864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:32.813321 26864 hybrid_clock.cc:648] HybridClock initialized: now 1714685132813275 us; error 36 us; skew 500 ppm
I20240502 21:25:32.813593 26864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:32.819468 26864 webserver.cc:460] Webserver started at http://127.24.226.190:34021/ using document root <none> and password file <none>
I20240502 21:25:32.820035 26864 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:32.820127 26864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:32.820385 26864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:32.822643 26864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "44e11de0e1d54d67b03329b705b8e888"
format_stamp: "Formatted at 2024-05-02 21:25:32 on dist-test-slave-41k1"
I20240502 21:25:32.823304 26864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "44e11de0e1d54d67b03329b705b8e888"
format_stamp: "Formatted at 2024-05-02 21:25:32 on dist-test-slave-41k1"
I20240502 21:25:32.827805 26864 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20240502 21:25:32.830814 26878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:32.831895 26864 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:32.832043 26864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/wal
uuid: "44e11de0e1d54d67b03329b705b8e888"
format_stamp: "Formatted at 2024-05-02 21:25:32 on dist-test-slave-41k1"
I20240502 21:25:32.832213 26864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:32.852015 26864 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:32.852753 26864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:32.853001 26864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:32.869227 26864 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.190:43225
I20240502 21:25:32.869254 26930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.190:43225 every 8 connection(s)
I20240502 21:25:32.870384 26864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 21:25:32.873566 26931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:32.875233 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 26864
I20240502 21:25:32.875442 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 21:25:32.883234 26931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44e11de0e1d54d67b03329b705b8e888: Bootstrap starting.
I20240502 21:25:32.886397 26931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44e11de0e1d54d67b03329b705b8e888: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:32.887324 26931 log.cc:824] T 00000000000000000000000000000000 P 44e11de0e1d54d67b03329b705b8e888: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:32.889609 26931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44e11de0e1d54d67b03329b705b8e888: No bootstrap required, opened a new log
I20240502 21:25:32.894891 26931 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44e11de0e1d54d67b03329b705b8e888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44e11de0e1d54d67b03329b705b8e888" member_type: VOTER last_known_addr { host: "127.24.226.190" port: 43225 } }
I20240502 21:25:32.895191 26931 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44e11de0e1d54d67b03329b705b8e888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:32.895284 26931 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44e11de0e1d54d67b03329b705b8e888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44e11de0e1d54d67b03329b705b8e888, State: Initialized, Role: FOLLOWER
I20240502 21:25:32.895721 26931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44e11de0e1d54d67b03329b705b8e888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44e11de0e1d54d67b03329b705b8e888" member_type: VOTER last_known_addr { host: "127.24.226.190" port: 43225 } }
I20240502 21:25:32.895884 26931 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44e11de0e1d54d67b03329b705b8e888 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:32.895963 26931 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44e11de0e1d54d67b03329b705b8e888 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:32.896068 26931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44e11de0e1d54d67b03329b705b8e888 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:32.897846 26931 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44e11de0e1d54d67b03329b705b8e888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44e11de0e1d54d67b03329b705b8e888" member_type: VOTER last_known_addr { host: "127.24.226.190" port: 43225 } }
I20240502 21:25:32.898262 26931 leader_election.cc:302] T 00000000000000000000000000000000 P 44e11de0e1d54d67b03329b705b8e888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44e11de0e1d54d67b03329b705b8e888; no voters:
I20240502 21:25:32.898792 26931 leader_election.cc:288] T 00000000000000000000000000000000 P 44e11de0e1d54d67b03329b705b8e888 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:32.898916 26936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44e11de0e1d54d67b03329b705b8e888 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:32.899528 26936 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44e11de0e1d54d67b03329b705b8e888 [term 1 LEADER]: Becoming Leader. State: Replica: 44e11de0e1d54d67b03329b705b8e888, State: Running, Role: LEADER
I20240502 21:25:32.899973 26936 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44e11de0e1d54d67b03329b705b8e888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44e11de0e1d54d67b03329b705b8e888" member_type: VOTER last_known_addr { host: "127.24.226.190" port: 43225 } }
I20240502 21:25:32.900403 26931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44e11de0e1d54d67b03329b705b8e888 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:25:32.902562 26937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44e11de0e1d54d67b03329b705b8e888 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44e11de0e1d54d67b03329b705b8e888" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44e11de0e1d54d67b03329b705b8e888" member_type: VOTER last_known_addr { host: "127.24.226.190" port: 43225 } } }
I20240502 21:25:32.903110 26937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44e11de0e1d54d67b03329b705b8e888 [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:32.902562 26938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44e11de0e1d54d67b03329b705b8e888 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44e11de0e1d54d67b03329b705b8e888. Latest consensus state: current_term: 1 leader_uuid: "44e11de0e1d54d67b03329b705b8e888" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44e11de0e1d54d67b03329b705b8e888" member_type: VOTER last_known_addr { host: "127.24.226.190" port: 43225 } } }
I20240502 21:25:32.904032 26938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44e11de0e1d54d67b03329b705b8e888 [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:32.908347 26941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:25:32.914098 26941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:25:32.922888 26941 catalog_manager.cc:1347] Generated new cluster ID: 2ec77aa220fc4a67bc4af39bea97c7e6
I20240502 21:25:32.922983 26941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:25:32.942083 26941 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:25:32.943434 26941 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:25:32.956287 26941 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 44e11de0e1d54d67b03329b705b8e888: Generated new TSK 0
I20240502 21:25:32.956960 26941 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:25:32.962196 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.129:0
--local_ip_for_outbound_sockets=127.24.226.129
--webserver_interface=127.24.226.129
--webserver_port=0
--tserver_master_addrs=127.24.226.190:43225
--builtin_ntp_servers=127.24.226.148:41147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 21:25:33.257211 26955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:33.257459 26955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:33.257551 26955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:33.263965 26955 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:33.264051 26955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:33.264145 26955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.129
I20240502 21:25:33.271824 26955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:41147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.24.226.129
--webserver_port=0
--tserver_master_addrs=127.24.226.190:43225
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.129
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:33.272859 26955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:33.274410 26955 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:33.284684 26960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:33.284659 26961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:33.284860 26963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:33.285452 26955 server_base.cc:1034] running on GCE node
I20240502 21:25:33.286016 26955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:33.286878 26955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:33.288084 26955 hybrid_clock.cc:648] HybridClock initialized: now 1714685133288016 us; error 60 us; skew 500 ppm
I20240502 21:25:33.288364 26955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:33.290367 26955 webserver.cc:460] Webserver started at http://127.24.226.129:38389/ using document root <none> and password file <none>
I20240502 21:25:33.290952 26955 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:33.291044 26955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:33.291357 26955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:33.293618 26955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "403577a6a03d42e180f5068a34f7a8e3"
format_stamp: "Formatted at 2024-05-02 21:25:33 on dist-test-slave-41k1"
I20240502 21:25:33.294306 26955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "403577a6a03d42e180f5068a34f7a8e3"
format_stamp: "Formatted at 2024-05-02 21:25:33 on dist-test-slave-41k1"
I20240502 21:25:33.298868 26955 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20240502 21:25:33.301872 26969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:33.302994 26955 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240502 21:25:33.303136 26955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "403577a6a03d42e180f5068a34f7a8e3"
format_stamp: "Formatted at 2024-05-02 21:25:33 on dist-test-slave-41k1"
I20240502 21:25:33.303297 26955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:33.324556 26955 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:33.325243 26955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:33.325464 26955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:33.326251 26955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:33.327497 26955 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:33.327577 26955 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:33.327721 26955 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:33.327800 26955 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:33.356916 26955 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.129:42533
I20240502 21:25:33.356985 27082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.129:42533 every 8 connection(s)
I20240502 21:25:33.358071 26955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 21:25:33.359125 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 26955
I20240502 21:25:33.359310 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 21:25:33.362789 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.130:0
--local_ip_for_outbound_sockets=127.24.226.130
--webserver_interface=127.24.226.130
--webserver_port=0
--tserver_master_addrs=127.24.226.190:43225
--builtin_ntp_servers=127.24.226.148:41147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:25:33.370824 27083 heartbeater.cc:347] Connected to a master server at 127.24.226.190:43225
I20240502 21:25:33.371080 27083 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:33.371819 27083 heartbeater.cc:511] Master 127.24.226.190:43225 requested a full tablet report, sending...
I20240502 21:25:33.373543 26895 ts_manager.cc:194] Registered new tserver with Master: 403577a6a03d42e180f5068a34f7a8e3 (127.24.226.129:42533)
I20240502 21:25:33.374907 26895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.129:46243
W20240502 21:25:33.655639 27086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:33.655853 27086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:33.655927 27086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:33.662163 27086 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:33.662251 27086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:33.662336 27086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.130
I20240502 21:25:33.669824 27086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:41147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.24.226.130
--webserver_port=0
--tserver_master_addrs=127.24.226.190:43225
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.130
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:33.670775 27086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:33.672272 27086 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:33.682312 27091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:33.682473 27094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:33.682518 27092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:33.683169 27086 server_base.cc:1034] running on GCE node
I20240502 21:25:33.683691 27086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:33.684494 27086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:33.685691 27086 hybrid_clock.cc:648] HybridClock initialized: now 1714685133685663 us; error 38 us; skew 500 ppm
I20240502 21:25:33.685937 27086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:33.687908 27086 webserver.cc:460] Webserver started at http://127.24.226.130:44729/ using document root <none> and password file <none>
I20240502 21:25:33.688596 27086 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:33.688714 27086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:33.689047 27086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:33.693409 27086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "44e2b21dcd4a494390b2460b34a13ebf"
format_stamp: "Formatted at 2024-05-02 21:25:33 on dist-test-slave-41k1"
I20240502 21:25:33.694089 27086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "44e2b21dcd4a494390b2460b34a13ebf"
format_stamp: "Formatted at 2024-05-02 21:25:33 on dist-test-slave-41k1"
I20240502 21:25:33.698377 27086 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 21:25:33.701282 27100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:33.702384 27086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:25:33.702517 27086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "44e2b21dcd4a494390b2460b34a13ebf"
format_stamp: "Formatted at 2024-05-02 21:25:33 on dist-test-slave-41k1"
I20240502 21:25:33.702684 27086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:33.715165 27086 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:33.715822 27086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:33.716039 27086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:33.716826 27086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:33.718075 27086 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:33.718183 27086 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:33.718297 27086 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:33.718370 27086 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:33.745616 27086 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.130:38725
I20240502 21:25:33.745664 27213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.130:38725 every 8 connection(s)
I20240502 21:25:33.746608 27086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 21:25:33.749311 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 27086
I20240502 21:25:33.749492 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 21:25:33.752130 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.131:0
--local_ip_for_outbound_sockets=127.24.226.131
--webserver_interface=127.24.226.131
--webserver_port=0
--tserver_master_addrs=127.24.226.190:43225
--builtin_ntp_servers=127.24.226.148:41147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:25:33.758018 27214 heartbeater.cc:347] Connected to a master server at 127.24.226.190:43225
I20240502 21:25:33.758301 27214 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:33.759151 27214 heartbeater.cc:511] Master 127.24.226.190:43225 requested a full tablet report, sending...
I20240502 21:25:33.760386 26895 ts_manager.cc:194] Registered new tserver with Master: 44e2b21dcd4a494390b2460b34a13ebf (127.24.226.130:38725)
I20240502 21:25:33.761090 26895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.130:39007
W20240502 21:25:34.041896 27217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:34.042153 27217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:34.042277 27217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:34.049082 27217 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:34.049230 27217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:34.049381 27217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.131
I20240502 21:25:34.056749 27217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:41147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.24.226.131
--webserver_port=0
--tserver_master_addrs=127.24.226.190:43225
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.131
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:34.057884 27217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:34.059463 27217 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:34.069559 27225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:34.069592 27222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:34.069634 27223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:34.070514 27217 server_base.cc:1034] running on GCE node
I20240502 21:25:34.070966 27217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:34.071738 27217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:34.072907 27217 hybrid_clock.cc:648] HybridClock initialized: now 1714685134072866 us; error 41 us; skew 500 ppm
I20240502 21:25:34.073158 27217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:34.075623 27217 webserver.cc:460] Webserver started at http://127.24.226.131:37393/ using document root <none> and password file <none>
I20240502 21:25:34.076210 27217 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:34.076301 27217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:34.076617 27217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:34.078920 27217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "3c4481a5e3e0403fb486bc97ac625507"
format_stamp: "Formatted at 2024-05-02 21:25:34 on dist-test-slave-41k1"
I20240502 21:25:34.079638 27217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "3c4481a5e3e0403fb486bc97ac625507"
format_stamp: "Formatted at 2024-05-02 21:25:34 on dist-test-slave-41k1"
I20240502 21:25:34.083971 27217 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20240502 21:25:34.086861 27231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:34.088104 27217 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:34.088268 27217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "3c4481a5e3e0403fb486bc97ac625507"
format_stamp: "Formatted at 2024-05-02 21:25:34 on dist-test-slave-41k1"
I20240502 21:25:34.088474 27217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:34.119673 27217 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:34.120337 27217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:34.120602 27217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:34.121395 27217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:34.122644 27217 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:34.122746 27217 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:34.122848 27217 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:34.122933 27217 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:34.150897 27217 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.131:40937
I20240502 21:25:34.150951 27344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.131:40937 every 8 connection(s)
I20240502 21:25:34.151968 27217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 21:25:34.158937 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 27217
I20240502 21:25:34.159138 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 21:25:34.162561 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.132:0
--local_ip_for_outbound_sockets=127.24.226.132
--webserver_interface=127.24.226.132
--webserver_port=0
--tserver_master_addrs=127.24.226.190:43225
--builtin_ntp_servers=127.24.226.148:41147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:25:34.164034 27345 heartbeater.cc:347] Connected to a master server at 127.24.226.190:43225
I20240502 21:25:34.164302 27345 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:34.165010 27345 heartbeater.cc:511] Master 127.24.226.190:43225 requested a full tablet report, sending...
I20240502 21:25:34.166296 26895 ts_manager.cc:194] Registered new tserver with Master: 3c4481a5e3e0403fb486bc97ac625507 (127.24.226.131:40937)
I20240502 21:25:34.166993 26895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.131:45477
I20240502 21:25:34.377241 27083 heartbeater.cc:503] Master 127.24.226.190:43225 was elected leader, sending a full tablet report...
W20240502 21:25:34.455475 27348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:34.455700 27348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:34.455798 27348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:34.462239 27348 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:34.462325 27348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:34.462451 27348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.132
I20240502 21:25:34.470566 27348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:41147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.24.226.132
--webserver_port=0
--tserver_master_addrs=127.24.226.190:43225
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.132
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:34.471544 27348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:34.472963 27348 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:34.482772 27354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:34.482815 27353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:34.482841 27356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:34.483511 27348 server_base.cc:1034] running on GCE node
I20240502 21:25:34.483959 27348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:34.484771 27348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:34.485957 27348 hybrid_clock.cc:648] HybridClock initialized: now 1714685134485928 us; error 28 us; skew 500 ppm
I20240502 21:25:34.486213 27348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:34.488024 27348 webserver.cc:460] Webserver started at http://127.24.226.132:34759/ using document root <none> and password file <none>
I20240502 21:25:34.488765 27348 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:34.488888 27348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:34.489177 27348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:34.491204 27348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "709816c490e547afad40c3a47ce3e6fb"
format_stamp: "Formatted at 2024-05-02 21:25:34 on dist-test-slave-41k1"
I20240502 21:25:34.491824 27348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "709816c490e547afad40c3a47ce3e6fb"
format_stamp: "Formatted at 2024-05-02 21:25:34 on dist-test-slave-41k1"
I20240502 21:25:34.495949 27348 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20240502 21:25:34.498754 27362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:34.499816 27348 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:34.499951 27348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "709816c490e547afad40c3a47ce3e6fb"
format_stamp: "Formatted at 2024-05-02 21:25:34 on dist-test-slave-41k1"
I20240502 21:25:34.500160 27348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:34.518232 27348 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:34.518910 27348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:34.519150 27348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:34.519956 27348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:34.521245 27348 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:34.521335 27348 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:34.521440 27348 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:34.521508 27348 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:34.551506 27348 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.132:42417
I20240502 21:25:34.551571 27474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.132:42417 every 8 connection(s)
I20240502 21:25:34.552628 27348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 21:25:34.558018 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 27348
I20240502 21:25:34.558205 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 21:25:34.560868 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.133:0
--local_ip_for_outbound_sockets=127.24.226.133
--webserver_interface=127.24.226.133
--webserver_port=0
--tserver_master_addrs=127.24.226.190:43225
--builtin_ntp_servers=127.24.226.148:41147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:25:34.564168 27475 heartbeater.cc:347] Connected to a master server at 127.24.226.190:43225
I20240502 21:25:34.564414 27475 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:34.565112 27475 heartbeater.cc:511] Master 127.24.226.190:43225 requested a full tablet report, sending...
I20240502 21:25:34.566294 26895 ts_manager.cc:194] Registered new tserver with Master: 709816c490e547afad40c3a47ce3e6fb (127.24.226.132:42417)
I20240502 21:25:34.567304 26895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.132:38111
I20240502 21:25:34.763716 27214 heartbeater.cc:503] Master 127.24.226.190:43225 was elected leader, sending a full tablet report...
W20240502 21:25:34.845556 27478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:34.845773 27478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:34.845871 27478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:34.852411 27478 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:34.852519 27478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:34.852656 27478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.133
I20240502 21:25:34.860291 27478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:41147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.24.226.133
--webserver_port=0
--tserver_master_addrs=127.24.226.190:43225
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.133
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:34.861307 27478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:34.862764 27478 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:34.874696 27484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:34.875373 27486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:34.874728 27483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:34.876299 27478 server_base.cc:1034] running on GCE node
I20240502 21:25:34.876791 27478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:34.877517 27478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:34.878712 27478 hybrid_clock.cc:648] HybridClock initialized: now 1714685134878668 us; error 47 us; skew 500 ppm
I20240502 21:25:34.879024 27478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:34.880926 27478 webserver.cc:460] Webserver started at http://127.24.226.133:40883/ using document root <none> and password file <none>
I20240502 21:25:34.881505 27478 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:34.881600 27478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:34.881870 27478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:34.883946 27478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "d84510593f28424bb0b23ed72d3b1351"
format_stamp: "Formatted at 2024-05-02 21:25:34 on dist-test-slave-41k1"
I20240502 21:25:34.884626 27478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "d84510593f28424bb0b23ed72d3b1351"
format_stamp: "Formatted at 2024-05-02 21:25:34 on dist-test-slave-41k1"
I20240502 21:25:34.889029 27478 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240502 21:25:34.891835 27492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:34.893076 27478 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:25:34.893244 27478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "d84510593f28424bb0b23ed72d3b1351"
format_stamp: "Formatted at 2024-05-02 21:25:34 on dist-test-slave-41k1"
I20240502 21:25:34.893414 27478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:34.911240 27478 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:34.911903 27478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:34.912143 27478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:34.913034 27478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:34.914361 27478 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:34.914449 27478 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:34.914549 27478 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:34.914618 27478 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:34.943239 27478 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.133:37421
I20240502 21:25:34.943291 27605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.133:37421 every 8 connection(s)
I20240502 21:25:34.944346 27478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:25:34.946298 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 27478
I20240502 21:25:34.946461 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 21:25:34.955582 27606 heartbeater.cc:347] Connected to a master server at 127.24.226.190:43225
I20240502 21:25:34.955814 27606 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:34.956537 27606 heartbeater.cc:511] Master 127.24.226.190:43225 requested a full tablet report, sending...
I20240502 21:25:34.957662 26895 ts_manager.cc:194] Registered new tserver with Master: d84510593f28424bb0b23ed72d3b1351 (127.24.226.133:37421)
I20240502 21:25:34.958417 26895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.133:52013
I20240502 21:25:34.961752 25482 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 21:25:34.978230 26895 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50176:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 21:25:35.010226 27410 tablet_service.cc:1467] Processing CreateTablet for tablet edff68a8054243f9bd1ae273f836be85 (DEFAULT_TABLE table=TestTable [id=ab686d5771be4eefb0a5213e6774161a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:35.011582 27410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edff68a8054243f9bd1ae273f836be85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:35.013614 27279 tablet_service.cc:1467] Processing CreateTablet for tablet edff68a8054243f9bd1ae273f836be85 (DEFAULT_TABLE table=TestTable [id=ab686d5771be4eefb0a5213e6774161a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:35.014819 27279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edff68a8054243f9bd1ae273f836be85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:35.016009 27017 tablet_service.cc:1467] Processing CreateTablet for tablet edff68a8054243f9bd1ae273f836be85 (DEFAULT_TABLE table=TestTable [id=ab686d5771be4eefb0a5213e6774161a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:35.017494 27017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edff68a8054243f9bd1ae273f836be85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:35.021813 27624 tablet_bootstrap.cc:492] T edff68a8054243f9bd1ae273f836be85 P 709816c490e547afad40c3a47ce3e6fb: Bootstrap starting.
I20240502 21:25:35.025285 27624 tablet_bootstrap.cc:654] T edff68a8054243f9bd1ae273f836be85 P 709816c490e547afad40c3a47ce3e6fb: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:35.026248 27624 log.cc:824] T edff68a8054243f9bd1ae273f836be85 P 709816c490e547afad40c3a47ce3e6fb: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:35.026824 27626 tablet_bootstrap.cc:492] T edff68a8054243f9bd1ae273f836be85 P 3c4481a5e3e0403fb486bc97ac625507: Bootstrap starting.
I20240502 21:25:35.029137 27628 tablet_bootstrap.cc:492] T edff68a8054243f9bd1ae273f836be85 P 403577a6a03d42e180f5068a34f7a8e3: Bootstrap starting.
I20240502 21:25:35.029740 27624 tablet_bootstrap.cc:492] T edff68a8054243f9bd1ae273f836be85 P 709816c490e547afad40c3a47ce3e6fb: No bootstrap required, opened a new log
I20240502 21:25:35.030014 27624 ts_tablet_manager.cc:1367] T edff68a8054243f9bd1ae273f836be85 P 709816c490e547afad40c3a47ce3e6fb: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 21:25:35.031318 27626 tablet_bootstrap.cc:654] T edff68a8054243f9bd1ae273f836be85 P 3c4481a5e3e0403fb486bc97ac625507: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:35.031733 27628 tablet_bootstrap.cc:654] T edff68a8054243f9bd1ae273f836be85 P 403577a6a03d42e180f5068a34f7a8e3: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:35.032481 27626 log.cc:824] T edff68a8054243f9bd1ae273f836be85 P 3c4481a5e3e0403fb486bc97ac625507: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:35.032601 27628 log.cc:824] T edff68a8054243f9bd1ae273f836be85 P 403577a6a03d42e180f5068a34f7a8e3: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:35.035243 27626 tablet_bootstrap.cc:492] T edff68a8054243f9bd1ae273f836be85 P 3c4481a5e3e0403fb486bc97ac625507: No bootstrap required, opened a new log
I20240502 21:25:35.035280 27628 tablet_bootstrap.cc:492] T edff68a8054243f9bd1ae273f836be85 P 403577a6a03d42e180f5068a34f7a8e3: No bootstrap required, opened a new log
I20240502 21:25:35.035504 27628 ts_tablet_manager.cc:1367] T edff68a8054243f9bd1ae273f836be85 P 403577a6a03d42e180f5068a34f7a8e3: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20240502 21:25:35.035794 27626 ts_tablet_manager.cc:1367] T edff68a8054243f9bd1ae273f836be85 P 3c4481a5e3e0403fb486bc97ac625507: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240502 21:25:35.037966 27624 raft_consensus.cc:348] T edff68a8054243f9bd1ae273f836be85 P 709816c490e547afad40c3a47ce3e6fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709816c490e547afad40c3a47ce3e6fb" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42417 } } peers { permanent_uuid: "403577a6a03d42e180f5068a34f7a8e3" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 42533 } } peers { permanent_uuid: "3c4481a5e3e0403fb486bc97ac625507" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40937 } }
I20240502 21:25:35.038376 27624 raft_consensus.cc:374] T edff68a8054243f9bd1ae273f836be85 P 709816c490e547afad40c3a47ce3e6fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:35.038486 27624 raft_consensus.cc:729] T edff68a8054243f9bd1ae273f836be85 P 709816c490e547afad40c3a47ce3e6fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 709816c490e547afad40c3a47ce3e6fb, State: Initialized, Role: FOLLOWER
I20240502 21:25:35.039155 27624 consensus_queue.cc:260] T edff68a8054243f9bd1ae273f836be85 P 709816c490e547afad40c3a47ce3e6fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709816c490e547afad40c3a47ce3e6fb" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42417 } } peers { permanent_uuid: "403577a6a03d42e180f5068a34f7a8e3" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 42533 } } peers { permanent_uuid: "3c4481a5e3e0403fb486bc97ac625507" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40937 } }
I20240502 21:25:35.040122 27475 heartbeater.cc:503] Master 127.24.226.190:43225 was elected leader, sending a full tablet report...
I20240502 21:25:35.041139 27624 ts_tablet_manager.cc:1398] T edff68a8054243f9bd1ae273f836be85 P 709816c490e547afad40c3a47ce3e6fb: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20240502 21:25:35.041752 27628 raft_consensus.cc:348] T edff68a8054243f9bd1ae273f836be85 P 403577a6a03d42e180f5068a34f7a8e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709816c490e547afad40c3a47ce3e6fb" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42417 } } peers { permanent_uuid: "403577a6a03d42e180f5068a34f7a8e3" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 42533 } } peers { permanent_uuid: "3c4481a5e3e0403fb486bc97ac625507" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40937 } }
I20240502 21:25:35.042104 27628 raft_consensus.cc:374] T edff68a8054243f9bd1ae273f836be85 P 403577a6a03d42e180f5068a34f7a8e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:35.042207 27628 raft_consensus.cc:729] T edff68a8054243f9bd1ae273f836be85 P 403577a6a03d42e180f5068a34f7a8e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 403577a6a03d42e180f5068a34f7a8e3, State: Initialized, Role: FOLLOWER
I20240502 21:25:35.042799 27628 consensus_queue.cc:260] T edff68a8054243f9bd1ae273f836be85 P 403577a6a03d42e180f5068a34f7a8e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709816c490e547afad40c3a47ce3e6fb" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42417 } } peers { permanent_uuid: "403577a6a03d42e180f5068a34f7a8e3" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 42533 } } peers { permanent_uuid: "3c4481a5e3e0403fb486bc97ac625507" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40937 } }
I20240502 21:25:35.043555 27626 raft_consensus.cc:348] T edff68a8054243f9bd1ae273f836be85 P 3c4481a5e3e0403fb486bc97ac625507 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709816c490e547afad40c3a47ce3e6fb" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42417 } } peers { permanent_uuid: "403577a6a03d42e180f5068a34f7a8e3" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 42533 } } peers { permanent_uuid: "3c4481a5e3e0403fb486bc97ac625507" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40937 } }
I20240502 21:25:35.043969 27626 raft_consensus.cc:374] T edff68a8054243f9bd1ae273f836be85 P 3c4481a5e3e0403fb486bc97ac625507 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:35.044083 27626 raft_consensus.cc:729] T edff68a8054243f9bd1ae273f836be85 P 3c4481a5e3e0403fb486bc97ac625507 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c4481a5e3e0403fb486bc97ac625507, State: Initialized, Role: FOLLOWER
I20240502 21:25:35.044610 27628 ts_tablet_manager.cc:1398] T edff68a8054243f9bd1ae273f836be85 P 403577a6a03d42e180f5068a34f7a8e3: Time spent starting tablet: real 0.009s user 0.006s sys 0.005s
I20240502 21:25:35.044735 27626 consensus_queue.cc:260] T edff68a8054243f9bd1ae273f836be85 P 3c4481a5e3e0403fb486bc97ac625507 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709816c490e547afad40c3a47ce3e6fb" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42417 } } peers { permanent_uuid: "403577a6a03d42e180f5068a34f7a8e3" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 42533 } } peers { permanent_uuid: "3c4481a5e3e0403fb486bc97ac625507" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40937 } }
I20240502 21:25:35.045693 27345 heartbeater.cc:503] Master 127.24.226.190:43225 was elected leader, sending a full tablet report...
I20240502 21:25:35.046221 27626 ts_tablet_manager.cc:1398] T edff68a8054243f9bd1ae273f836be85 P 3c4481a5e3e0403fb486bc97ac625507: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
W20240502 21:25:35.054147 27476 tablet.cc:2369] T edff68a8054243f9bd1ae273f836be85 P 709816c490e547afad40c3a47ce3e6fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 21:25:35.110569 27084 tablet.cc:2369] T edff68a8054243f9bd1ae273f836be85 P 403577a6a03d42e180f5068a34f7a8e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 21:25:35.153923 27346 tablet.cc:2369] T edff68a8054243f9bd1ae273f836be85 P 3c4481a5e3e0403fb486bc97ac625507: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:25:35.230949 27631 raft_consensus.cc:482] T edff68a8054243f9bd1ae273f836be85 P 709816c490e547afad40c3a47ce3e6fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:35.231178 27631 raft_consensus.cc:504] T edff68a8054243f9bd1ae273f836be85 P 709816c490e547afad40c3a47ce3e6fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709816c490e547afad40c3a47ce3e6fb" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42417 } } peers { permanent_uuid: "403577a6a03d42e180f5068a34f7a8e3" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 42533 } } peers { permanent_uuid: "3c4481a5e3e0403fb486bc97ac625507" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40937 } }
I20240502 21:25:35.232729 27631 leader_election.cc:288] T edff68a8054243f9bd1ae273f836be85 P 709816c490e547afad40c3a47ce3e6fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 403577a6a03d42e180f5068a34f7a8e3 (127.24.226.129:42533), 3c4481a5e3e0403fb486bc97ac625507 (127.24.226.131:40937)
I20240502 21:25:35.239096 27299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edff68a8054243f9bd1ae273f836be85" candidate_uuid: "709816c490e547afad40c3a47ce3e6fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4481a5e3e0403fb486bc97ac625507" is_pre_election: true
I20240502 21:25:35.239104 27037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edff68a8054243f9bd1ae273f836be85" candidate_uuid: "709816c490e547afad40c3a47ce3e6fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "403577a6a03d42e180f5068a34f7a8e3" is_pre_election: true
I20240502 21:25:35.239480 27299 raft_consensus.cc:2454] T edff68a8054243f9bd1ae273f836be85 P 3c4481a5e3e0403fb486bc97ac625507 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 709816c490e547afad40c3a47ce3e6fb in term 0.
I20240502 21:25:35.239503 27037 raft_consensus.cc:2454] T edff68a8054243f9bd1ae273f836be85 P 403577a6a03d42e180f5068a34f7a8e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 709816c490e547afad40c3a47ce3e6fb in term 0.
I20240502 21:25:35.239984 27363 leader_election.cc:302] T edff68a8054243f9bd1ae273f836be85 P 709816c490e547afad40c3a47ce3e6fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c4481a5e3e0403fb486bc97ac625507, 709816c490e547afad40c3a47ce3e6fb; no voters:
I20240502 21:25:35.240309 27631 raft_consensus.cc:2781] T edff68a8054243f9bd1ae273f836be85 P 709816c490e547afad40c3a47ce3e6fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:35.240494 27631 raft_consensus.cc:482] T edff68a8054243f9bd1ae273f836be85 P 709816c490e547afad40c3a47ce3e6fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:35.240597 27631 raft_consensus.cc:3037] T edff68a8054243f9bd1ae273f836be85 P 709816c490e547afad40c3a47ce3e6fb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:35.242453 27631 raft_consensus.cc:504] T edff68a8054243f9bd1ae273f836be85 P 709816c490e547afad40c3a47ce3e6fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709816c490e547afad40c3a47ce3e6fb" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42417 } } peers { permanent_uuid: "403577a6a03d42e180f5068a34f7a8e3" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 42533 } } peers { permanent_uuid: "3c4481a5e3e0403fb486bc97ac625507" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40937 } }
I20240502 21:25:35.242935 27631 leader_election.cc:288] T edff68a8054243f9bd1ae273f836be85 P 709816c490e547afad40c3a47ce3e6fb [CANDIDATE]: Term 1 election: Requested vote from peers 403577a6a03d42e180f5068a34f7a8e3 (127.24.226.129:42533), 3c4481a5e3e0403fb486bc97ac625507 (127.24.226.131:40937)
I20240502 21:25:35.243261 27037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edff68a8054243f9bd1ae273f836be85" candidate_uuid: "709816c490e547afad40c3a47ce3e6fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "403577a6a03d42e180f5068a34f7a8e3"
I20240502 21:25:35.243410 27037 raft_consensus.cc:3037] T edff68a8054243f9bd1ae273f836be85 P 403577a6a03d42e180f5068a34f7a8e3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:35.243409 27299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edff68a8054243f9bd1ae273f836be85" candidate_uuid: "709816c490e547afad40c3a47ce3e6fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c4481a5e3e0403fb486bc97ac625507"
I20240502 21:25:35.243557 27299 raft_consensus.cc:3037] T edff68a8054243f9bd1ae273f836be85 P 3c4481a5e3e0403fb486bc97ac625507 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:35.245715 27037 raft_consensus.cc:2454] T edff68a8054243f9bd1ae273f836be85 P 403577a6a03d42e180f5068a34f7a8e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 709816c490e547afad40c3a47ce3e6fb in term 1.
I20240502 21:25:35.245956 27299 raft_consensus.cc:2454] T edff68a8054243f9bd1ae273f836be85 P 3c4481a5e3e0403fb486bc97ac625507 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 709816c490e547afad40c3a47ce3e6fb in term 1.
I20240502 21:25:35.246102 27366 leader_election.cc:302] T edff68a8054243f9bd1ae273f836be85 P 709816c490e547afad40c3a47ce3e6fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 403577a6a03d42e180f5068a34f7a8e3, 709816c490e547afad40c3a47ce3e6fb; no voters:
I20240502 21:25:35.246409 27631 raft_consensus.cc:2781] T edff68a8054243f9bd1ae273f836be85 P 709816c490e547afad40c3a47ce3e6fb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:35.246778 27631 raft_consensus.cc:686] T edff68a8054243f9bd1ae273f836be85 P 709816c490e547afad40c3a47ce3e6fb [term 1 LEADER]: Becoming Leader. State: Replica: 709816c490e547afad40c3a47ce3e6fb, State: Running, Role: LEADER
I20240502 21:25:35.247195 27631 consensus_queue.cc:237] T edff68a8054243f9bd1ae273f836be85 P 709816c490e547afad40c3a47ce3e6fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709816c490e547afad40c3a47ce3e6fb" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42417 } } peers { permanent_uuid: "403577a6a03d42e180f5068a34f7a8e3" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 42533 } } peers { permanent_uuid: "3c4481a5e3e0403fb486bc97ac625507" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40937 } }
I20240502 21:25:35.251559 26895 catalog_manager.cc:5484] T edff68a8054243f9bd1ae273f836be85 P 709816c490e547afad40c3a47ce3e6fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 709816c490e547afad40c3a47ce3e6fb (127.24.226.132). New cstate: current_term: 1 leader_uuid: "709816c490e547afad40c3a47ce3e6fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709816c490e547afad40c3a47ce3e6fb" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 42417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "403577a6a03d42e180f5068a34f7a8e3" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 42533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c4481a5e3e0403fb486bc97ac625507" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40937 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:35.306219 25482 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 21:25:35.308048 25482 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 403577a6a03d42e180f5068a34f7a8e3 to finish bootstrapping
I20240502 21:25:35.315392 25482 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 44e2b21dcd4a494390b2460b34a13ebf to finish bootstrapping
I20240502 21:25:35.322922 25482 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 3c4481a5e3e0403fb486bc97ac625507 to finish bootstrapping
I20240502 21:25:35.328554 25482 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 709816c490e547afad40c3a47ce3e6fb to finish bootstrapping
I20240502 21:25:35.333977 25482 ts_itest-base.cc:233] Waiting for 0 tablets on tserver d84510593f28424bb0b23ed72d3b1351 to finish bootstrapping
I20240502 21:25:35.341428 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 26955
I20240502 21:25:35.342116 27078 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:35.420845 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 26955
W20240502 21:25:35.625638 27366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.226.129:42533: connect: Connection refused (error 111)
W20240502 21:25:35.626996 27366 consensus_peers.cc:485] T edff68a8054243f9bd1ae273f836be85 P 709816c490e547afad40c3a47ce3e6fb -> Peer 403577a6a03d42e180f5068a34f7a8e3 (127.24.226.129:42533): Couldn't send request to peer 403577a6a03d42e180f5068a34f7a8e3. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.129:42533: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:35.841419 27279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:35.842938 27148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:35.844023 27410 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:35.852931 27631 consensus_queue.cc:1035] T edff68a8054243f9bd1ae273f836be85 P 709816c490e547afad40c3a47ce3e6fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c4481a5e3e0403fb486bc97ac625507" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:35.867921 27540 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:35.960345 27606 heartbeater.cc:503] Master 127.24.226.190:43225 was elected leader, sending a full tablet report...
I20240502 21:25:35.984148 27279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:35.988183 27148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:35.991766 27410 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:36.002349 27540 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:36.130340 27279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:36.135933 27410 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:36.144562 27148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:36.144729 27540 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:36.681139 27279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:36.687150 27148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:36.687238 27410 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:36.692696 27540 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:36.796622 27410 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:36.798688 27279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:36.807574 27148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:36.807806 27540 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:36.968408 27279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:36.973512 27410 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:36.975477 27148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:36.978726 27540 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:37.107460 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 27086
I20240502 21:25:37.108840 27209 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:37.194427 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 27086
I20240502 21:25:37.211588 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 27217
I20240502 21:25:37.212661 27339 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:37.300663 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 27217
I20240502 21:25:37.318686 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 27348
I20240502 21:25:37.319628 27469 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:37.409680 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 27348
I20240502 21:25:37.428221 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 27478
I20240502 21:25:37.429211 27600 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:37.515733 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 27478
I20240502 21:25:37.532603 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 26864
I20240502 21:25:37.533623 26926 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:37.642819 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 26864
2024-05-02T21:25:37Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/1 (5212 ms)
[----------] 1 test from RebalanceStartCriteriaTest (5212 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/1
I20240502 21:25:37.684059 25482 test_util.cc:268] Using random seed: 1049134663
I20240502 21:25:37.685719 25482 ts_itest-base.cc:112] Starting cluster with:
I20240502 21:25:37.685772 25482 ts_itest-base.cc:113] --------------
I20240502 21:25:37.685823 25482 ts_itest-base.cc:114] 7 tablet servers
I20240502 21:25:37.685871 25482 ts_itest-base.cc:115] 1 replicas per TS
I20240502 21:25:37.685931 25482 ts_itest-base.cc:116] --------------
2024-05-02T21:25:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T21:25:37Z Disabled control of system clock
I20240502 21:25:37.707911 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.226.190:32867
--webserver_interface=127.24.226.190
--webserver_port=0
--builtin_ntp_servers=127.24.226.148:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.226.190:32867
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240502 21:25:37.997604 27847 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 21:25:37.997888 27847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:37.998005 27847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:37.998114 27847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:38.004842 27847 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:38.004940 27847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 21:25:38.005028 27847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:38.005084 27847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 21:25:38.005165 27847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 21:25:38.013154 27847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.24.226.190:32867
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.226.190:32867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.24.226.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:38.014154 27847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:38.015609 27847 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:38.025332 27855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:38.025372 27852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:38.025391 27853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:38.026122 27847 server_base.cc:1034] running on GCE node
I20240502 21:25:38.026715 27847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:38.027848 27847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:38.029043 27847 hybrid_clock.cc:648] HybridClock initialized: now 1714685138028989 us; error 47 us; skew 500 ppm
I20240502 21:25:38.029349 27847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:38.035936 27847 webserver.cc:460] Webserver started at http://127.24.226.190:42195/ using document root <none> and password file <none>
I20240502 21:25:38.036671 27847 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:38.036811 27847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:38.037084 27847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:38.039650 27847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "5dc89f2545c14a249317777eb9782c83"
format_stamp: "Formatted at 2024-05-02 21:25:38 on dist-test-slave-41k1"
I20240502 21:25:38.040354 27847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "5dc89f2545c14a249317777eb9782c83"
format_stamp: "Formatted at 2024-05-02 21:25:38 on dist-test-slave-41k1"
I20240502 21:25:38.045238 27847 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240502 21:25:38.048316 27861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:38.049572 27847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 21:25:38.049731 27847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/wal
uuid: "5dc89f2545c14a249317777eb9782c83"
format_stamp: "Formatted at 2024-05-02 21:25:38 on dist-test-slave-41k1"
I20240502 21:25:38.049894 27847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:38.083974 27847 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:38.084707 27847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:38.084951 27847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:38.099772 27847 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.190:32867
I20240502 21:25:38.099810 27913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.190:32867 every 8 connection(s)
I20240502 21:25:38.100946 27847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 21:25:38.102591 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 27847
I20240502 21:25:38.102762 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 21:25:38.104566 27914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:38.116791 27914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dc89f2545c14a249317777eb9782c83: Bootstrap starting.
I20240502 21:25:38.119371 27914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5dc89f2545c14a249317777eb9782c83: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:38.120512 27914 log.cc:824] T 00000000000000000000000000000000 P 5dc89f2545c14a249317777eb9782c83: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:38.122597 27914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dc89f2545c14a249317777eb9782c83: No bootstrap required, opened a new log
I20240502 21:25:38.128908 27914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5dc89f2545c14a249317777eb9782c83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc89f2545c14a249317777eb9782c83" member_type: VOTER last_known_addr { host: "127.24.226.190" port: 32867 } }
I20240502 21:25:38.129184 27914 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5dc89f2545c14a249317777eb9782c83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:38.129273 27914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5dc89f2545c14a249317777eb9782c83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dc89f2545c14a249317777eb9782c83, State: Initialized, Role: FOLLOWER
I20240502 21:25:38.129784 27914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5dc89f2545c14a249317777eb9782c83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc89f2545c14a249317777eb9782c83" member_type: VOTER last_known_addr { host: "127.24.226.190" port: 32867 } }
I20240502 21:25:38.129966 27914 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5dc89f2545c14a249317777eb9782c83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:38.130048 27914 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5dc89f2545c14a249317777eb9782c83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:38.130168 27914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5dc89f2545c14a249317777eb9782c83 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:38.131820 27914 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5dc89f2545c14a249317777eb9782c83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc89f2545c14a249317777eb9782c83" member_type: VOTER last_known_addr { host: "127.24.226.190" port: 32867 } }
I20240502 21:25:38.132200 27914 leader_election.cc:302] T 00000000000000000000000000000000 P 5dc89f2545c14a249317777eb9782c83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5dc89f2545c14a249317777eb9782c83; no voters:
I20240502 21:25:38.132743 27914 leader_election.cc:288] T 00000000000000000000000000000000 P 5dc89f2545c14a249317777eb9782c83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:38.132845 27919 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5dc89f2545c14a249317777eb9782c83 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:38.133440 27919 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5dc89f2545c14a249317777eb9782c83 [term 1 LEADER]: Becoming Leader. State: Replica: 5dc89f2545c14a249317777eb9782c83, State: Running, Role: LEADER
I20240502 21:25:38.133919 27919 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5dc89f2545c14a249317777eb9782c83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc89f2545c14a249317777eb9782c83" member_type: VOTER last_known_addr { host: "127.24.226.190" port: 32867 } }
I20240502 21:25:38.134493 27914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5dc89f2545c14a249317777eb9782c83 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:25:38.136638 27921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dc89f2545c14a249317777eb9782c83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dc89f2545c14a249317777eb9782c83. Latest consensus state: current_term: 1 leader_uuid: "5dc89f2545c14a249317777eb9782c83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc89f2545c14a249317777eb9782c83" member_type: VOTER last_known_addr { host: "127.24.226.190" port: 32867 } } }
I20240502 21:25:38.136916 27921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dc89f2545c14a249317777eb9782c83 [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:38.137506 27920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dc89f2545c14a249317777eb9782c83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5dc89f2545c14a249317777eb9782c83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc89f2545c14a249317777eb9782c83" member_type: VOTER last_known_addr { host: "127.24.226.190" port: 32867 } } }
I20240502 21:25:38.137734 27920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dc89f2545c14a249317777eb9782c83 [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:38.138455 27925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:25:38.142958 27925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:25:38.150239 27925 catalog_manager.cc:1347] Generated new cluster ID: 54c6acc0783b4799ada61b3309468cbd
I20240502 21:25:38.150326 27925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:25:38.161089 27925 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:25:38.162137 27925 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:25:38.171664 27925 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 5dc89f2545c14a249317777eb9782c83: Generated new TSK 0
I20240502 21:25:38.172451 27925 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:25:38.188123 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.129:0
--local_ip_for_outbound_sockets=127.24.226.129
--webserver_interface=127.24.226.129
--webserver_port=0
--tserver_master_addrs=127.24.226.190:32867
--builtin_ntp_servers=127.24.226.148:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 21:25:38.498939 27938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:38.499194 27938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:38.499327 27938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:38.505661 27938 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:38.505755 27938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:38.505898 27938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.129
I20240502 21:25:38.513816 27938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.24.226.129
--webserver_port=0
--tserver_master_addrs=127.24.226.190:32867
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.129
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:38.514863 27938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:38.516507 27938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:38.526907 27943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:38.526991 27944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:38.526949 27946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:38.528095 27938 server_base.cc:1034] running on GCE node
I20240502 21:25:38.528674 27938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:38.529543 27938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:38.530715 27938 hybrid_clock.cc:648] HybridClock initialized: now 1714685138530676 us; error 39 us; skew 500 ppm
I20240502 21:25:38.530988 27938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:38.532891 27938 webserver.cc:460] Webserver started at http://127.24.226.129:43655/ using document root <none> and password file <none>
I20240502 21:25:38.533553 27938 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:38.533649 27938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:38.533924 27938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:38.536239 27938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "e0444722e0404faf9e70415e169e464f"
format_stamp: "Formatted at 2024-05-02 21:25:38 on dist-test-slave-41k1"
I20240502 21:25:38.536973 27938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "e0444722e0404faf9e70415e169e464f"
format_stamp: "Formatted at 2024-05-02 21:25:38 on dist-test-slave-41k1"
I20240502 21:25:38.541438 27938 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 21:25:38.544310 27952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:38.545487 27938 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:25:38.545615 27938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e0444722e0404faf9e70415e169e464f"
format_stamp: "Formatted at 2024-05-02 21:25:38 on dist-test-slave-41k1"
I20240502 21:25:38.545783 27938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:38.562948 27938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:38.563580 27938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:38.563810 27938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:38.564608 27938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:38.565860 27938 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:38.565938 27938 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:38.566017 27938 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:38.566082 27938 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:38.595070 27938 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.129:39509
I20240502 21:25:38.595181 28065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.129:39509 every 8 connection(s)
I20240502 21:25:38.596186 27938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 21:25:38.605489 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 27938
I20240502 21:25:38.605664 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 21:25:38.607223 28066 heartbeater.cc:347] Connected to a master server at 127.24.226.190:32867
I20240502 21:25:38.607471 28066 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:38.608273 28066 heartbeater.cc:511] Master 127.24.226.190:32867 requested a full tablet report, sending...
I20240502 21:25:38.608803 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.130:0
--local_ip_for_outbound_sockets=127.24.226.130
--webserver_interface=127.24.226.130
--webserver_port=0
--tserver_master_addrs=127.24.226.190:32867
--builtin_ntp_servers=127.24.226.148:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:25:38.610226 27878 ts_manager.cc:194] Registered new tserver with Master: e0444722e0404faf9e70415e169e464f (127.24.226.129:39509)
I20240502 21:25:38.611625 27878 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.129:43737
W20240502 21:25:38.896564 28069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:38.896806 28069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:38.896919 28069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:38.903278 28069 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:38.903370 28069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:38.903532 28069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.130
I20240502 21:25:38.911033 28069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.24.226.130
--webserver_port=0
--tserver_master_addrs=127.24.226.190:32867
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.130
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:38.911999 28069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:38.913486 28069 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:38.923333 28074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:38.923544 28077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:38.923758 28075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:38.924235 28069 server_base.cc:1034] running on GCE node
I20240502 21:25:38.924711 28069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:38.925398 28069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:38.926576 28069 hybrid_clock.cc:648] HybridClock initialized: now 1714685138926532 us; error 40 us; skew 500 ppm
I20240502 21:25:38.926831 28069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:38.928918 28069 webserver.cc:460] Webserver started at http://127.24.226.130:46125/ using document root <none> and password file <none>
I20240502 21:25:38.929574 28069 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:38.929670 28069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:38.929944 28069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:38.933820 28069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "37e9c08ca35f419b874cf64812057324"
format_stamp: "Formatted at 2024-05-02 21:25:38 on dist-test-slave-41k1"
I20240502 21:25:38.934463 28069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "37e9c08ca35f419b874cf64812057324"
format_stamp: "Formatted at 2024-05-02 21:25:38 on dist-test-slave-41k1"
I20240502 21:25:38.938987 28069 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 21:25:38.941754 28083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:38.942832 28069 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 21:25:38.942968 28069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "37e9c08ca35f419b874cf64812057324"
format_stamp: "Formatted at 2024-05-02 21:25:38 on dist-test-slave-41k1"
I20240502 21:25:38.943138 28069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:38.957175 28069 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:38.957813 28069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:38.958048 28069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:38.958911 28069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:38.960192 28069 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:38.960289 28069 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:38.960389 28069 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:38.960628 28069 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240502 21:25:38.988355 28069 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.130:35721
I20240502 21:25:38.988411 28195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.130:35721 every 8 connection(s)
I20240502 21:25:38.989519 28069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 21:25:38.994450 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 28069
I20240502 21:25:38.994604 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 21:25:38.997284 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.131:0
--local_ip_for_outbound_sockets=127.24.226.131
--webserver_interface=127.24.226.131
--webserver_port=0
--tserver_master_addrs=127.24.226.190:32867
--builtin_ntp_servers=127.24.226.148:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:25:38.999986 28196 heartbeater.cc:347] Connected to a master server at 127.24.226.190:32867
I20240502 21:25:39.000213 28196 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:39.000968 28196 heartbeater.cc:511] Master 127.24.226.190:32867 requested a full tablet report, sending...
I20240502 21:25:39.002372 27878 ts_manager.cc:194] Registered new tserver with Master: 37e9c08ca35f419b874cf64812057324 (127.24.226.130:35721)
I20240502 21:25:39.003304 27878 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.130:49195
W20240502 21:25:39.287575 28199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:39.287793 28199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:39.287897 28199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:39.294512 28199 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:39.294610 28199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:39.294757 28199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.131
I20240502 21:25:39.302597 28199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.24.226.131
--webserver_port=0
--tserver_master_addrs=127.24.226.190:32867
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.131
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:39.303682 28199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:39.305222 28199 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:39.315712 28205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:39.315778 28208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:39.315758 28206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:39.316351 28199 server_base.cc:1034] running on GCE node
I20240502 21:25:39.316867 28199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:39.317608 28199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:39.318817 28199 hybrid_clock.cc:648] HybridClock initialized: now 1714685139318779 us; error 26 us; skew 500 ppm
I20240502 21:25:39.319144 28199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:39.321162 28199 webserver.cc:460] Webserver started at http://127.24.226.131:41055/ using document root <none> and password file <none>
I20240502 21:25:39.321789 28199 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:39.321892 28199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:39.322183 28199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:39.324353 28199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e5b2432ac32b4c8281bfe645caf7aea2"
format_stamp: "Formatted at 2024-05-02 21:25:39 on dist-test-slave-41k1"
I20240502 21:25:39.325083 28199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e5b2432ac32b4c8281bfe645caf7aea2"
format_stamp: "Formatted at 2024-05-02 21:25:39 on dist-test-slave-41k1"
I20240502 21:25:39.329538 28199 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 21:25:39.332505 28214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:39.333603 28199 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 21:25:39.333737 28199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e5b2432ac32b4c8281bfe645caf7aea2"
format_stamp: "Formatted at 2024-05-02 21:25:39 on dist-test-slave-41k1"
I20240502 21:25:39.333901 28199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:39.355505 28199 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:39.356168 28199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:39.356391 28199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:39.357159 28199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:39.358440 28199 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:39.358538 28199 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:39.358654 28199 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:39.358726 28199 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:39.387220 28199 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.131:46635
I20240502 21:25:39.387272 28326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.131:46635 every 8 connection(s)
I20240502 21:25:39.388337 28199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 21:25:39.394407 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 28199
I20240502 21:25:39.394603 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 21:25:39.397486 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.132:0
--local_ip_for_outbound_sockets=127.24.226.132
--webserver_interface=127.24.226.132
--webserver_port=0
--tserver_master_addrs=127.24.226.190:32867
--builtin_ntp_servers=127.24.226.148:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:25:39.399217 28327 heartbeater.cc:347] Connected to a master server at 127.24.226.190:32867
I20240502 21:25:39.399468 28327 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:39.400094 28327 heartbeater.cc:511] Master 127.24.226.190:32867 requested a full tablet report, sending...
I20240502 21:25:39.401647 27878 ts_manager.cc:194] Registered new tserver with Master: e5b2432ac32b4c8281bfe645caf7aea2 (127.24.226.131:46635)
I20240502 21:25:39.402621 27878 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.131:47101
I20240502 21:25:39.613970 28066 heartbeater.cc:503] Master 127.24.226.190:32867 was elected leader, sending a full tablet report...
W20240502 21:25:39.681545 28330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:39.681763 28330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:39.681835 28330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:39.688035 28330 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:39.688163 28330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:39.688331 28330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.132
I20240502 21:25:39.695761 28330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.24.226.132
--webserver_port=0
--tserver_master_addrs=127.24.226.190:32867
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.132
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:39.696794 28330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:39.698307 28330 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:39.708483 28336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:39.708531 28338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:39.708563 28335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:39.709160 28330 server_base.cc:1034] running on GCE node
I20240502 21:25:39.709674 28330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:39.710443 28330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:39.711611 28330 hybrid_clock.cc:648] HybridClock initialized: now 1714685139711562 us; error 43 us; skew 500 ppm
I20240502 21:25:39.711876 28330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:39.713836 28330 webserver.cc:460] Webserver started at http://127.24.226.132:38319/ using document root <none> and password file <none>
I20240502 21:25:39.714442 28330 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:39.714541 28330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:39.714833 28330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:39.717067 28330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f"
format_stamp: "Formatted at 2024-05-02 21:25:39 on dist-test-slave-41k1"
I20240502 21:25:39.717733 28330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f"
format_stamp: "Formatted at 2024-05-02 21:25:39 on dist-test-slave-41k1"
I20240502 21:25:39.722009 28330 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20240502 21:25:39.724874 28344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:39.725963 28330 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 21:25:39.726099 28330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f"
format_stamp: "Formatted at 2024-05-02 21:25:39 on dist-test-slave-41k1"
I20240502 21:25:39.726269 28330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:39.746606 28330 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:39.747231 28330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:39.747453 28330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:39.748298 28330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:39.749658 28330 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:39.749737 28330 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:39.749806 28330 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:39.749855 28330 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:39.776921 28330 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.132:38475
I20240502 21:25:39.776983 28457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.132:38475 every 8 connection(s)
I20240502 21:25:39.778048 28330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 21:25:39.783810 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 28330
I20240502 21:25:39.783982 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 21:25:39.787328 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.133:0
--local_ip_for_outbound_sockets=127.24.226.133
--webserver_interface=127.24.226.133
--webserver_port=0
--tserver_master_addrs=127.24.226.190:32867
--builtin_ntp_servers=127.24.226.148:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:25:39.792462 28458 heartbeater.cc:347] Connected to a master server at 127.24.226.190:32867
I20240502 21:25:39.792708 28458 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:39.793403 28458 heartbeater.cc:511] Master 127.24.226.190:32867 requested a full tablet report, sending...
I20240502 21:25:39.794677 27878 ts_manager.cc:194] Registered new tserver with Master: 7b66fa9f55994d0caf5dc7d5a8189b2f (127.24.226.132:38475)
I20240502 21:25:39.795818 27878 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.132:49693
I20240502 21:25:40.005316 28196 heartbeater.cc:503] Master 127.24.226.190:32867 was elected leader, sending a full tablet report...
W20240502 21:25:40.082089 28461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:40.082329 28461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:40.082448 28461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:40.089165 28461 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:40.089265 28461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:40.089408 28461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.133
I20240502 21:25:40.096781 28461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.24.226.133
--webserver_port=0
--tserver_master_addrs=127.24.226.190:32867
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.133
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:40.097792 28461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:40.099500 28461 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:40.109804 28469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:40.109831 28467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:40.109982 28466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:40.110574 28461 server_base.cc:1034] running on GCE node
I20240502 21:25:40.111073 28461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:40.111852 28461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:40.113049 28461 hybrid_clock.cc:648] HybridClock initialized: now 1714685140113002 us; error 44 us; skew 500 ppm
I20240502 21:25:40.113337 28461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:40.115301 28461 webserver.cc:460] Webserver started at http://127.24.226.133:43505/ using document root <none> and password file <none>
I20240502 21:25:40.115900 28461 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:40.116006 28461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:40.116274 28461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:40.118525 28461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "538c626600cd419bb5b9eeae5ded7610"
format_stamp: "Formatted at 2024-05-02 21:25:40 on dist-test-slave-41k1"
I20240502 21:25:40.119228 28461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "538c626600cd419bb5b9eeae5ded7610"
format_stamp: "Formatted at 2024-05-02 21:25:40 on dist-test-slave-41k1"
I20240502 21:25:40.123847 28461 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20240502 21:25:40.127024 28475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:40.128201 28461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 21:25:40.128356 28461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "538c626600cd419bb5b9eeae5ded7610"
format_stamp: "Formatted at 2024-05-02 21:25:40 on dist-test-slave-41k1"
I20240502 21:25:40.128583 28461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:40.149572 28461 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:40.150207 28461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:40.150439 28461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:40.151237 28461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:40.152520 28461 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:40.152607 28461 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:40.152705 28461 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:40.152771 28461 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:40.182370 28461 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.133:38491
I20240502 21:25:40.182430 28587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.133:38491 every 8 connection(s)
I20240502 21:25:40.183388 28461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:25:40.184217 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 28461
I20240502 21:25:40.184468 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 21:25:40.187465 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.134:0
--local_ip_for_outbound_sockets=127.24.226.134
--webserver_interface=127.24.226.134
--webserver_port=0
--tserver_master_addrs=127.24.226.190:32867
--builtin_ntp_servers=127.24.226.148:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:25:40.195008 28588 heartbeater.cc:347] Connected to a master server at 127.24.226.190:32867
I20240502 21:25:40.195369 28588 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:40.196056 28588 heartbeater.cc:511] Master 127.24.226.190:32867 requested a full tablet report, sending...
I20240502 21:25:40.197273 27878 ts_manager.cc:194] Registered new tserver with Master: 538c626600cd419bb5b9eeae5ded7610 (127.24.226.133:38491)
I20240502 21:25:40.197978 27878 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.133:45881
I20240502 21:25:40.404906 28327 heartbeater.cc:503] Master 127.24.226.190:32867 was elected leader, sending a full tablet report...
W20240502 21:25:40.475785 28591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:40.475997 28591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:40.476070 28591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:40.482427 28591 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:40.482513 28591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:40.482606 28591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.134
I20240502 21:25:40.490336 28591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.24.226.134
--webserver_port=0
--tserver_master_addrs=127.24.226.190:32867
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.134
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:40.491254 28591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:40.492728 28591 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:40.502776 28596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:40.502904 28599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:40.503268 28597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:40.503798 28591 server_base.cc:1034] running on GCE node
I20240502 21:25:40.504236 28591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:40.504997 28591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:40.506201 28591 hybrid_clock.cc:648] HybridClock initialized: now 1714685140506161 us; error 28 us; skew 500 ppm
I20240502 21:25:40.506515 28591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:40.508535 28591 webserver.cc:460] Webserver started at http://127.24.226.134:39983/ using document root <none> and password file <none>
I20240502 21:25:40.509171 28591 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:40.509263 28591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:40.509542 28591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:40.511679 28591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "6174b3a926654ec291571611f865116d"
format_stamp: "Formatted at 2024-05-02 21:25:40 on dist-test-slave-41k1"
I20240502 21:25:40.512284 28591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "6174b3a926654ec291571611f865116d"
format_stamp: "Formatted at 2024-05-02 21:25:40 on dist-test-slave-41k1"
I20240502 21:25:40.516717 28591 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 21:25:40.519567 28605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:40.520620 28591 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:25:40.520748 28591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6174b3a926654ec291571611f865116d"
format_stamp: "Formatted at 2024-05-02 21:25:40 on dist-test-slave-41k1"
I20240502 21:25:40.520901 28591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:40.550473 28591 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:40.551151 28591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:40.551476 28591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:40.553483 28591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:40.556082 28591 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:40.556161 28591 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:40.556268 28591 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:40.556329 28591 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:40.584695 28591 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.134:37323
I20240502 21:25:40.584754 28717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.134:37323 every 8 connection(s)
I20240502 21:25:40.585868 28591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 21:25:40.592798 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 28591
I20240502 21:25:40.592969 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 21:25:40.596226 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.135:0
--local_ip_for_outbound_sockets=127.24.226.135
--webserver_interface=127.24.226.135
--webserver_port=0
--tserver_master_addrs=127.24.226.190:32867
--builtin_ntp_servers=127.24.226.148:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:25:40.597523 28718 heartbeater.cc:347] Connected to a master server at 127.24.226.190:32867
I20240502 21:25:40.597802 28718 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:40.598755 28718 heartbeater.cc:511] Master 127.24.226.190:32867 requested a full tablet report, sending...
I20240502 21:25:40.600201 27878 ts_manager.cc:194] Registered new tserver with Master: 6174b3a926654ec291571611f865116d (127.24.226.134:37323)
I20240502 21:25:40.601154 27878 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.134:54109
I20240502 21:25:40.798005 28458 heartbeater.cc:503] Master 127.24.226.190:32867 was elected leader, sending a full tablet report...
W20240502 21:25:40.892709 28721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:40.892977 28721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:40.893064 28721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:40.899456 28721 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:40.899549 28721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:40.899645 28721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.135
I20240502 21:25:40.907207 28721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.24.226.135
--webserver_port=0
--tserver_master_addrs=127.24.226.190:32867
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.135
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:40.908185 28721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:40.909832 28721 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:40.920038 28729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:40.920217 28726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:40.920313 28727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:40.920922 28721 server_base.cc:1034] running on GCE node
I20240502 21:25:40.921422 28721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:40.922192 28721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:40.923370 28721 hybrid_clock.cc:648] HybridClock initialized: now 1714685140923332 us; error 31 us; skew 500 ppm
I20240502 21:25:40.923655 28721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:40.925626 28721 webserver.cc:460] Webserver started at http://127.24.226.135:43919/ using document root <none> and password file <none>
I20240502 21:25:40.926231 28721 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:40.926318 28721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:40.926565 28721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:40.928983 28721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "18d49d446e634bb5a630d06b99730d21"
format_stamp: "Formatted at 2024-05-02 21:25:40 on dist-test-slave-41k1"
I20240502 21:25:40.929656 28721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "18d49d446e634bb5a630d06b99730d21"
format_stamp: "Formatted at 2024-05-02 21:25:40 on dist-test-slave-41k1"
I20240502 21:25:40.933939 28721 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 21:25:40.936944 28735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:40.938110 28721 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:25:40.938262 28721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "18d49d446e634bb5a630d06b99730d21"
format_stamp: "Formatted at 2024-05-02 21:25:40 on dist-test-slave-41k1"
I20240502 21:25:40.938460 28721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:40.981254 28721 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:40.981873 28721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:40.982079 28721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:40.982817 28721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:40.984046 28721 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:40.984123 28721 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:40.984197 28721 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:40.984244 28721 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:41.014539 28721 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.135:33083
I20240502 21:25:41.014598 28847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.135:33083 every 8 connection(s)
I20240502 21:25:41.015697 28721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 21:25:41.023516 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 28721
I20240502 21:25:41.023706 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 21:25:41.027387 28848 heartbeater.cc:347] Connected to a master server at 127.24.226.190:32867
I20240502 21:25:41.027627 28848 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:41.028327 28848 heartbeater.cc:511] Master 127.24.226.190:32867 requested a full tablet report, sending...
I20240502 21:25:41.029595 27877 ts_manager.cc:194] Registered new tserver with Master: 18d49d446e634bb5a630d06b99730d21 (127.24.226.135:33083)
I20240502 21:25:41.030473 27877 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.135:34085
I20240502 21:25:41.039372 25482 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 21:25:41.054244 27877 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36142:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 21:25:41.078770 28000 tablet_service.cc:1467] Processing CreateTablet for tablet e18547d3903940c1ab603b8d4129b6d3 (DEFAULT_TABLE table=TestTable [id=c0e550d8c6444707a5179f3ec075781e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:41.079957 28000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e18547d3903940c1ab603b8d4129b6d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:41.088553 28862 tablet_bootstrap.cc:492] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f: Bootstrap starting.
I20240502 21:25:41.091269 28862 tablet_bootstrap.cc:654] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:41.092226 28862 log.cc:824] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:41.094108 28862 tablet_bootstrap.cc:492] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f: No bootstrap required, opened a new log
I20240502 21:25:41.094302 28862 ts_tablet_manager.cc:1367] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 21:25:41.100381 28862 raft_consensus.cc:348] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } }
I20240502 21:25:41.100672 28862 raft_consensus.cc:374] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:41.100800 28862 raft_consensus.cc:729] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0444722e0404faf9e70415e169e464f, State: Initialized, Role: FOLLOWER
I20240502 21:25:41.101312 28862 consensus_queue.cc:260] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } }
I20240502 21:25:41.101486 28862 raft_consensus.cc:388] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:41.101593 28862 raft_consensus.cc:482] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:41.101738 28862 raft_consensus.cc:3037] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:41.103484 28862 raft_consensus.cc:504] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } }
I20240502 21:25:41.103876 28862 leader_election.cc:302] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0444722e0404faf9e70415e169e464f; no voters:
I20240502 21:25:41.104388 28862 leader_election.cc:288] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:41.104525 28864 raft_consensus.cc:2781] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:41.105800 28862 ts_tablet_manager.cc:1398] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20240502 21:25:41.106148 28864 raft_consensus.cc:686] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Becoming Leader. State: Replica: e0444722e0404faf9e70415e169e464f, State: Running, Role: LEADER
I20240502 21:25:41.106587 28864 consensus_queue.cc:237] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } }
I20240502 21:25:41.111649 27877 catalog_manager.cc:5484] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f reported cstate change: term changed from 0 to 1, leader changed from <none> to e0444722e0404faf9e70415e169e464f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "e0444722e0404faf9e70415e169e464f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:41.121851 25482 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 21:25:41.123437 25482 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e0444722e0404faf9e70415e169e464f to finish bootstrapping
I20240502 21:25:41.130854 25482 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 37e9c08ca35f419b874cf64812057324 to finish bootstrapping
I20240502 21:25:41.138475 25482 ts_itest-base.cc:233] Waiting for 0 tablets on tserver e5b2432ac32b4c8281bfe645caf7aea2 to finish bootstrapping
I20240502 21:25:41.145716 25482 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 7b66fa9f55994d0caf5dc7d5a8189b2f to finish bootstrapping
I20240502 21:25:41.153123 25482 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 538c626600cd419bb5b9eeae5ded7610 to finish bootstrapping
I20240502 21:25:41.160046 25482 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6174b3a926654ec291571611f865116d to finish bootstrapping
I20240502 21:25:41.166648 25482 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 18d49d446e634bb5a630d06b99730d21 to finish bootstrapping
I20240502 21:25:41.174139 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 28199
I20240502 21:25:41.174960 28322 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:41.200129 28588 heartbeater.cc:503] Master 127.24.226.190:32867 was elected leader, sending a full tablet report...
I20240502 21:25:41.247159 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 28199
I20240502 21:25:41.262706 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 28330
I20240502 21:25:41.263676 28453 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:41.336544 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 28330
I20240502 21:25:41.353478 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 28461
I20240502 21:25:41.354569 28583 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:41.427531 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 28461
I20240502 21:25:41.442731 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 28591
I20240502 21:25:41.443776 28713 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:41.513298 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 28591
I20240502 21:25:41.527835 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 28721
I20240502 21:25:41.528868 28843 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:41.605710 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 28721
I20240502 21:25:45.375156 27877 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36142:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:25:45.384356 28000 tablet_service.cc:1467] Processing CreateTablet for tablet 92f8d60108924e42866b1f35c3673443 (DEFAULT_TABLE table=rebalance_test_table_0 [id=affadeda486b4ca9b4db96ff7f6c53cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:45.384824 28000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92f8d60108924e42866b1f35c3673443. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:45.390285 28887 tablet_bootstrap.cc:492] T 92f8d60108924e42866b1f35c3673443 P e0444722e0404faf9e70415e169e464f: Bootstrap starting.
I20240502 21:25:45.391682 28887 tablet_bootstrap.cc:654] T 92f8d60108924e42866b1f35c3673443 P e0444722e0404faf9e70415e169e464f: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:45.393321 28887 tablet_bootstrap.cc:492] T 92f8d60108924e42866b1f35c3673443 P e0444722e0404faf9e70415e169e464f: No bootstrap required, opened a new log
I20240502 21:25:45.392798 28131 tablet_service.cc:1467] Processing CreateTablet for tablet 53fea94f8310494b9354f09fc92ca06a (DEFAULT_TABLE table=rebalance_test_table_0 [id=affadeda486b4ca9b4db96ff7f6c53cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:45.393460 28887 ts_tablet_manager.cc:1367] T 92f8d60108924e42866b1f35c3673443 P e0444722e0404faf9e70415e169e464f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:25:45.392989 28130 tablet_service.cc:1467] Processing CreateTablet for tablet c3aa757477ad450ab8d4c3029b16c670 (DEFAULT_TABLE table=rebalance_test_table_0 [id=affadeda486b4ca9b4db96ff7f6c53cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:45.394094 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53fea94f8310494b9354f09fc92ca06a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:45.394284 28887 raft_consensus.cc:348] T 92f8d60108924e42866b1f35c3673443 P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } }
I20240502 21:25:45.394412 28887 raft_consensus.cc:374] T 92f8d60108924e42866b1f35c3673443 P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:45.394505 28887 raft_consensus.cc:729] T 92f8d60108924e42866b1f35c3673443 P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0444722e0404faf9e70415e169e464f, State: Initialized, Role: FOLLOWER
I20240502 21:25:45.394534 28130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3aa757477ad450ab8d4c3029b16c670. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:45.394712 28887 consensus_queue.cc:260] T 92f8d60108924e42866b1f35c3673443 P e0444722e0404faf9e70415e169e464f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } }
I20240502 21:25:45.394857 28887 raft_consensus.cc:388] T 92f8d60108924e42866b1f35c3673443 P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:45.394927 28887 raft_consensus.cc:482] T 92f8d60108924e42866b1f35c3673443 P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:45.395015 28887 raft_consensus.cc:3037] T 92f8d60108924e42866b1f35c3673443 P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:45.396474 28887 raft_consensus.cc:504] T 92f8d60108924e42866b1f35c3673443 P e0444722e0404faf9e70415e169e464f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } }
I20240502 21:25:45.396673 28887 leader_election.cc:302] T 92f8d60108924e42866b1f35c3673443 P e0444722e0404faf9e70415e169e464f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0444722e0404faf9e70415e169e464f; no voters:
I20240502 21:25:45.397060 28887 leader_election.cc:288] T 92f8d60108924e42866b1f35c3673443 P e0444722e0404faf9e70415e169e464f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:45.397215 28889 raft_consensus.cc:2781] T 92f8d60108924e42866b1f35c3673443 P e0444722e0404faf9e70415e169e464f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:45.403790 28887 ts_tablet_manager.cc:1398] T 92f8d60108924e42866b1f35c3673443 P e0444722e0404faf9e70415e169e464f: Time spent starting tablet: real 0.010s user 0.004s sys 0.006s
I20240502 21:25:45.404662 28889 raft_consensus.cc:686] T 92f8d60108924e42866b1f35c3673443 P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Becoming Leader. State: Replica: e0444722e0404faf9e70415e169e464f, State: Running, Role: LEADER
I20240502 21:25:45.404886 28889 consensus_queue.cc:237] T 92f8d60108924e42866b1f35c3673443 P e0444722e0404faf9e70415e169e464f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } }
I20240502 21:25:45.406841 28893 tablet_bootstrap.cc:492] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324: Bootstrap starting.
I20240502 21:25:45.406927 27877 catalog_manager.cc:5484] T 92f8d60108924e42866b1f35c3673443 P e0444722e0404faf9e70415e169e464f reported cstate change: term changed from 0 to 1, leader changed from <none> to e0444722e0404faf9e70415e169e464f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "e0444722e0404faf9e70415e169e464f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:45.409911 28893 tablet_bootstrap.cc:654] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:45.410866 28893 log.cc:824] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:45.413038 28893 tablet_bootstrap.cc:492] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324: No bootstrap required, opened a new log
I20240502 21:25:45.413224 28893 ts_tablet_manager.cc:1367] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240502 21:25:45.418295 28893 raft_consensus.cc:348] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } }
I20240502 21:25:45.418634 28893 raft_consensus.cc:374] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:45.418743 28893 raft_consensus.cc:729] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37e9c08ca35f419b874cf64812057324, State: Initialized, Role: FOLLOWER
I20240502 21:25:45.419368 28893 consensus_queue.cc:260] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } }
I20240502 21:25:45.419587 28893 raft_consensus.cc:388] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:45.419719 28893 raft_consensus.cc:482] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:45.419883 28893 raft_consensus.cc:3037] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:45.421770 28893 raft_consensus.cc:504] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } }
I20240502 21:25:45.422155 28893 leader_election.cc:302] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37e9c08ca35f419b874cf64812057324; no voters:
I20240502 21:25:45.422686 28893 leader_election.cc:288] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:45.422827 28896 raft_consensus.cc:2781] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:45.424165 28896 raft_consensus.cc:686] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Becoming Leader. State: Replica: 37e9c08ca35f419b874cf64812057324, State: Running, Role: LEADER
I20240502 21:25:45.424286 28893 ts_tablet_manager.cc:1398] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20240502 21:25:45.424615 28893 tablet_bootstrap.cc:492] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324: Bootstrap starting.
I20240502 21:25:45.424631 28896 consensus_queue.cc:237] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } }
I20240502 21:25:45.425674 28893 tablet_bootstrap.cc:654] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:45.427450 28893 tablet_bootstrap.cc:492] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324: No bootstrap required, opened a new log
I20240502 21:25:45.427577 28893 ts_tablet_manager.cc:1367] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 21:25:45.428313 28893 raft_consensus.cc:348] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } }
I20240502 21:25:45.428493 28893 raft_consensus.cc:374] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:45.428625 28893 raft_consensus.cc:729] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37e9c08ca35f419b874cf64812057324, State: Initialized, Role: FOLLOWER
I20240502 21:25:45.428504 27877 catalog_manager.cc:5484] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37e9c08ca35f419b874cf64812057324 (127.24.226.130). New cstate: current_term: 1 leader_uuid: "37e9c08ca35f419b874cf64812057324" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:45.428930 28893 consensus_queue.cc:260] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } }
I20240502 21:25:45.429122 28893 raft_consensus.cc:388] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:45.429221 28893 raft_consensus.cc:482] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:45.429340 28893 raft_consensus.cc:3037] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:45.431268 28893 raft_consensus.cc:504] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } }
I20240502 21:25:45.431490 28893 leader_election.cc:302] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37e9c08ca35f419b874cf64812057324; no voters:
I20240502 21:25:45.431700 28893 leader_election.cc:288] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:45.431802 28898 raft_consensus.cc:2781] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:45.432153 28893 ts_tablet_manager.cc:1398] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240502 21:25:45.432984 28898 raft_consensus.cc:686] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Becoming Leader. State: Replica: 37e9c08ca35f419b874cf64812057324, State: Running, Role: LEADER
I20240502 21:25:45.433235 28898 consensus_queue.cc:237] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } }
I20240502 21:25:45.435173 27877 catalog_manager.cc:5484] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37e9c08ca35f419b874cf64812057324 (127.24.226.130). New cstate: current_term: 1 leader_uuid: "37e9c08ca35f419b874cf64812057324" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } health_report { overall_health: HEALTHY } } }
W20240502 21:25:45.738624 28904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:45.738910 28904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:45.745528 28904 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 106.364 ms
time per row: 0.0531818 ms
I20240502 21:25:45.980660 27875 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36142:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:25:45.988209 28000 tablet_service.cc:1467] Processing CreateTablet for tablet 0357275420cb49d799d26033f9804bef (DEFAULT_TABLE table=rebalance_test_table_1 [id=f3b24b58d96b4bceb7b44fc6fe502ad0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:45.988277 28131 tablet_service.cc:1467] Processing CreateTablet for tablet 4aeae3dbc5f7474dba8ed8841feab6c3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f3b24b58d96b4bceb7b44fc6fe502ad0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:45.988687 28000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0357275420cb49d799d26033f9804bef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:45.988696 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aeae3dbc5f7474dba8ed8841feab6c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:45.994760 28944 tablet_bootstrap.cc:492] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f: Bootstrap starting.
I20240502 21:25:45.994824 27999 tablet_service.cc:1467] Processing CreateTablet for tablet d3b7a019c07149d9a291267a76909669 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f3b24b58d96b4bceb7b44fc6fe502ad0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:45.995132 28945 tablet_bootstrap.cc:492] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324: Bootstrap starting.
I20240502 21:25:45.995275 27999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3b7a019c07149d9a291267a76909669. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:45.996495 28944 tablet_bootstrap.cc:654] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:45.996588 28945 tablet_bootstrap.cc:654] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:46.003587 28945 tablet_bootstrap.cc:492] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324: No bootstrap required, opened a new log
I20240502 21:25:46.003609 28944 tablet_bootstrap.cc:492] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f: No bootstrap required, opened a new log
I20240502 21:25:46.003708 28945 ts_tablet_manager.cc:1367] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20240502 21:25:46.003718 28944 ts_tablet_manager.cc:1367] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240502 21:25:46.004316 28944 raft_consensus.cc:348] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } }
I20240502 21:25:46.004384 28945 raft_consensus.cc:348] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } }
I20240502 21:25:46.004513 28944 raft_consensus.cc:374] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:46.004570 28945 raft_consensus.cc:374] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:46.004606 28944 raft_consensus.cc:729] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0444722e0404faf9e70415e169e464f, State: Initialized, Role: FOLLOWER
I20240502 21:25:46.004664 28945 raft_consensus.cc:729] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37e9c08ca35f419b874cf64812057324, State: Initialized, Role: FOLLOWER
I20240502 21:25:46.004940 28945 consensus_queue.cc:260] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } }
I20240502 21:25:46.004961 28944 consensus_queue.cc:260] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } }
I20240502 21:25:46.005113 28945 raft_consensus.cc:388] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:46.005124 28944 raft_consensus.cc:388] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:46.005195 28944 raft_consensus.cc:482] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:46.005198 28945 raft_consensus.cc:482] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:46.005287 28945 raft_consensus.cc:3037] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:46.005290 28944 raft_consensus.cc:3037] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:46.006974 28944 raft_consensus.cc:504] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } }
I20240502 21:25:46.007169 28944 leader_election.cc:302] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0444722e0404faf9e70415e169e464f; no voters:
I20240502 21:25:46.007324 28944 leader_election.cc:288] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:46.007375 28889 raft_consensus.cc:2781] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:46.007694 28945 raft_consensus.cc:504] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } }
I20240502 21:25:46.007908 28945 leader_election.cc:302] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37e9c08ca35f419b874cf64812057324; no voters:
I20240502 21:25:46.008131 28898 raft_consensus.cc:2781] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:46.008262 28898 raft_consensus.cc:686] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Becoming Leader. State: Replica: 37e9c08ca35f419b874cf64812057324, State: Running, Role: LEADER
I20240502 21:25:46.008483 28898 consensus_queue.cc:237] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } }
I20240502 21:25:46.012269 28945 leader_election.cc:288] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:46.012817 28945 ts_tablet_manager.cc:1398] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324: Time spent starting tablet: real 0.009s user 0.004s sys 0.001s
I20240502 21:25:46.019291 28944 ts_tablet_manager.cc:1398] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f: Time spent starting tablet: real 0.015s user 0.008s sys 0.008s
I20240502 21:25:46.019521 28889 raft_consensus.cc:686] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Becoming Leader. State: Replica: e0444722e0404faf9e70415e169e464f, State: Running, Role: LEADER
I20240502 21:25:46.019578 28944 tablet_bootstrap.cc:492] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f: Bootstrap starting.
I20240502 21:25:46.020140 28889 consensus_queue.cc:237] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } }
I20240502 21:25:46.021250 28944 tablet_bootstrap.cc:654] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:46.021508 27875 catalog_manager.cc:5484] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37e9c08ca35f419b874cf64812057324 (127.24.226.130). New cstate: current_term: 1 leader_uuid: "37e9c08ca35f419b874cf64812057324" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:46.023924 27877 catalog_manager.cc:5484] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f reported cstate change: term changed from 0 to 1, leader changed from <none> to e0444722e0404faf9e70415e169e464f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "e0444722e0404faf9e70415e169e464f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:46.025482 28944 tablet_bootstrap.cc:492] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f: No bootstrap required, opened a new log
I20240502 21:25:46.025622 28944 ts_tablet_manager.cc:1367] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20240502 21:25:46.027240 28944 raft_consensus.cc:348] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } }
I20240502 21:25:46.027396 28944 raft_consensus.cc:374] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:46.027467 28944 raft_consensus.cc:729] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0444722e0404faf9e70415e169e464f, State: Initialized, Role: FOLLOWER
I20240502 21:25:46.027961 28944 consensus_queue.cc:260] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } }
I20240502 21:25:46.028085 28944 raft_consensus.cc:388] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:46.028169 28944 raft_consensus.cc:482] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:46.028251 28944 raft_consensus.cc:3037] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:46.030341 28944 raft_consensus.cc:504] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } }
I20240502 21:25:46.030535 28944 leader_election.cc:302] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0444722e0404faf9e70415e169e464f; no voters:
I20240502 21:25:46.030714 28944 leader_election.cc:288] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:46.030771 28949 raft_consensus.cc:2781] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:46.030932 28949 raft_consensus.cc:686] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Becoming Leader. State: Replica: e0444722e0404faf9e70415e169e464f, State: Running, Role: LEADER
I20240502 21:25:46.031214 28944 ts_tablet_manager.cc:1398] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240502 21:25:46.031257 28949 consensus_queue.cc:237] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } }
I20240502 21:25:46.033218 27875 catalog_manager.cc:5484] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f reported cstate change: term changed from 0 to 1, leader changed from <none> to e0444722e0404faf9e70415e169e464f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "e0444722e0404faf9e70415e169e464f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } health_report { overall_health: HEALTHY } } }
W20240502 21:25:46.323873 28954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:46.324136 28954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:46.330787 28954 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 139.724 ms
time per row: 0.0698619 ms
I20240502 21:25:46.600332 27877 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36142:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:25:46.609057 28131 tablet_service.cc:1467] Processing CreateTablet for tablet e5fb7a69b82c42678d35301469f999ea (DEFAULT_TABLE table=rebalance_test_table_2 [id=6fc5508a149a46b6b19e75ddf5817ae1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:46.609083 28130 tablet_service.cc:1467] Processing CreateTablet for tablet c607e33cdf814d00b3d005493d941a0a (DEFAULT_TABLE table=rebalance_test_table_2 [id=6fc5508a149a46b6b19e75ddf5817ae1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:46.609474 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5fb7a69b82c42678d35301469f999ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:46.609702 28130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c607e33cdf814d00b3d005493d941a0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:46.610103 27999 tablet_service.cc:1467] Processing CreateTablet for tablet 6a7eaa487ef4426292192a0f4768567c (DEFAULT_TABLE table=rebalance_test_table_2 [id=6fc5508a149a46b6b19e75ddf5817ae1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:46.610424 27999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a7eaa487ef4426292192a0f4768567c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:46.624990 28996 tablet_bootstrap.cc:492] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324: Bootstrap starting.
I20240502 21:25:46.625192 28995 tablet_bootstrap.cc:492] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f: Bootstrap starting.
I20240502 21:25:46.626612 28996 tablet_bootstrap.cc:654] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:46.626627 28995 tablet_bootstrap.cc:654] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:46.631599 28995 tablet_bootstrap.cc:492] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f: No bootstrap required, opened a new log
I20240502 21:25:46.631762 28995 ts_tablet_manager.cc:1367] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240502 21:25:46.631908 28996 tablet_bootstrap.cc:492] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324: No bootstrap required, opened a new log
I20240502 21:25:46.632042 28996 ts_tablet_manager.cc:1367] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240502 21:25:46.632404 28995 raft_consensus.cc:348] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } }
I20240502 21:25:46.632565 28995 raft_consensus.cc:374] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:46.632648 28995 raft_consensus.cc:729] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0444722e0404faf9e70415e169e464f, State: Initialized, Role: FOLLOWER
I20240502 21:25:46.632687 28996 raft_consensus.cc:348] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } }
I20240502 21:25:46.632825 28996 raft_consensus.cc:374] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:46.632907 28996 raft_consensus.cc:729] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37e9c08ca35f419b874cf64812057324, State: Initialized, Role: FOLLOWER
I20240502 21:25:46.632841 28995 consensus_queue.cc:260] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } }
I20240502 21:25:46.633006 28995 raft_consensus.cc:388] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:46.633080 28995 raft_consensus.cc:482] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:46.633103 28996 consensus_queue.cc:260] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } }
I20240502 21:25:46.633181 28995 raft_consensus.cc:3037] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:46.633271 28996 raft_consensus.cc:388] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:46.633339 28996 raft_consensus.cc:482] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:46.633433 28996 raft_consensus.cc:3037] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:46.634819 28995 raft_consensus.cc:504] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } }
I20240502 21:25:46.634922 28996 raft_consensus.cc:504] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } }
I20240502 21:25:46.635021 28995 leader_election.cc:302] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0444722e0404faf9e70415e169e464f; no voters:
I20240502 21:25:46.635140 28996 leader_election.cc:302] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37e9c08ca35f419b874cf64812057324; no voters:
I20240502 21:25:46.635190 28995 leader_election.cc:288] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:46.635257 28949 raft_consensus.cc:2781] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:46.635300 28996 leader_election.cc:288] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:46.635370 28898 raft_consensus.cc:2781] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:46.635731 28995 ts_tablet_manager.cc:1398] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 21:25:46.636036 28898 raft_consensus.cc:686] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Becoming Leader. State: Replica: 37e9c08ca35f419b874cf64812057324, State: Running, Role: LEADER
I20240502 21:25:46.636309 28898 consensus_queue.cc:237] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } }
I20240502 21:25:46.637027 28996 ts_tablet_manager.cc:1398] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 21:25:46.636998 28949 raft_consensus.cc:686] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Becoming Leader. State: Replica: e0444722e0404faf9e70415e169e464f, State: Running, Role: LEADER
I20240502 21:25:46.637238 28949 consensus_queue.cc:237] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } }
I20240502 21:25:46.637312 28996 tablet_bootstrap.cc:492] T e5fb7a69b82c42678d35301469f999ea P 37e9c08ca35f419b874cf64812057324: Bootstrap starting.
I20240502 21:25:46.638911 28996 tablet_bootstrap.cc:654] T e5fb7a69b82c42678d35301469f999ea P 37e9c08ca35f419b874cf64812057324: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:46.639530 27877 catalog_manager.cc:5484] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37e9c08ca35f419b874cf64812057324 (127.24.226.130). New cstate: current_term: 1 leader_uuid: "37e9c08ca35f419b874cf64812057324" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:46.640206 27875 catalog_manager.cc:5484] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f reported cstate change: term changed from 0 to 1, leader changed from <none> to e0444722e0404faf9e70415e169e464f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "e0444722e0404faf9e70415e169e464f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:46.641005 28996 tablet_bootstrap.cc:492] T e5fb7a69b82c42678d35301469f999ea P 37e9c08ca35f419b874cf64812057324: No bootstrap required, opened a new log
I20240502 21:25:46.641142 28996 ts_tablet_manager.cc:1367] T e5fb7a69b82c42678d35301469f999ea P 37e9c08ca35f419b874cf64812057324: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:25:46.641785 28996 raft_consensus.cc:348] T e5fb7a69b82c42678d35301469f999ea P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } }
I20240502 21:25:46.641921 28996 raft_consensus.cc:374] T e5fb7a69b82c42678d35301469f999ea P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:46.642025 28996 raft_consensus.cc:729] T e5fb7a69b82c42678d35301469f999ea P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37e9c08ca35f419b874cf64812057324, State: Initialized, Role: FOLLOWER
I20240502 21:25:46.642274 28996 consensus_queue.cc:260] T e5fb7a69b82c42678d35301469f999ea P 37e9c08ca35f419b874cf64812057324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } }
I20240502 21:25:46.642488 28996 raft_consensus.cc:388] T e5fb7a69b82c42678d35301469f999ea P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:46.642588 28996 raft_consensus.cc:482] T e5fb7a69b82c42678d35301469f999ea P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:46.642669 28996 raft_consensus.cc:3037] T e5fb7a69b82c42678d35301469f999ea P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:46.644873 28996 raft_consensus.cc:504] T e5fb7a69b82c42678d35301469f999ea P 37e9c08ca35f419b874cf64812057324 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } }
I20240502 21:25:46.645040 28996 leader_election.cc:302] T e5fb7a69b82c42678d35301469f999ea P 37e9c08ca35f419b874cf64812057324 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37e9c08ca35f419b874cf64812057324; no voters:
I20240502 21:25:46.645207 28996 leader_election.cc:288] T e5fb7a69b82c42678d35301469f999ea P 37e9c08ca35f419b874cf64812057324 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:46.645243 28898 raft_consensus.cc:2781] T e5fb7a69b82c42678d35301469f999ea P 37e9c08ca35f419b874cf64812057324 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:46.645375 28898 raft_consensus.cc:686] T e5fb7a69b82c42678d35301469f999ea P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Becoming Leader. State: Replica: 37e9c08ca35f419b874cf64812057324, State: Running, Role: LEADER
I20240502 21:25:46.645614 28898 consensus_queue.cc:237] T e5fb7a69b82c42678d35301469f999ea P 37e9c08ca35f419b874cf64812057324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } }
I20240502 21:25:46.645694 28996 ts_tablet_manager.cc:1398] T e5fb7a69b82c42678d35301469f999ea P 37e9c08ca35f419b874cf64812057324: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240502 21:25:46.647862 27877 catalog_manager.cc:5484] T e5fb7a69b82c42678d35301469f999ea P 37e9c08ca35f419b874cf64812057324 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37e9c08ca35f419b874cf64812057324 (127.24.226.130). New cstate: current_term: 1 leader_uuid: "37e9c08ca35f419b874cf64812057324" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } health_report { overall_health: HEALTHY } } }
W20240502 21:25:46.938638 29006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:46.938941 29006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:46.945196 29006 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 102.028 ms
time per row: 0.0510142 ms
I20240502 21:25:47.171207 27871 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36142:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:25:47.178546 27999 tablet_service.cc:1467] Processing CreateTablet for tablet 96f3f6d4adfc423e8c2f2ef24ba0cf1f (DEFAULT_TABLE table=rebalance_test_table_3 [id=22e3b2f360454a519b85e699fa7dd2d4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:47.178859 28130 tablet_service.cc:1467] Processing CreateTablet for tablet fe34567313ce48c1a0caeb2b306bce57 (DEFAULT_TABLE table=rebalance_test_table_3 [id=22e3b2f360454a519b85e699fa7dd2d4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:47.179039 27999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96f3f6d4adfc423e8c2f2ef24ba0cf1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:47.179263 28130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe34567313ce48c1a0caeb2b306bce57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:47.179173 28000 tablet_service.cc:1467] Processing CreateTablet for tablet 25affb8859f94f308ff433d9cb288de0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=22e3b2f360454a519b85e699fa7dd2d4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:47.179528 28000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25affb8859f94f308ff433d9cb288de0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:47.185298 29044 tablet_bootstrap.cc:492] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f: Bootstrap starting.
I20240502 21:25:47.185537 29045 tablet_bootstrap.cc:492] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324: Bootstrap starting.
I20240502 21:25:47.186520 29044 tablet_bootstrap.cc:654] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:47.186619 29045 tablet_bootstrap.cc:654] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:47.188086 29044 tablet_bootstrap.cc:492] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f: No bootstrap required, opened a new log
I20240502 21:25:47.188187 29045 tablet_bootstrap.cc:492] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324: No bootstrap required, opened a new log
I20240502 21:25:47.188208 29044 ts_tablet_manager.cc:1367] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:25:47.188323 29045 ts_tablet_manager.cc:1367] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:25:47.188838 29044 raft_consensus.cc:348] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } }
I20240502 21:25:47.188997 29044 raft_consensus.cc:374] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:47.188954 29045 raft_consensus.cc:348] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } }
I20240502 21:25:47.189096 29044 raft_consensus.cc:729] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0444722e0404faf9e70415e169e464f, State: Initialized, Role: FOLLOWER
I20240502 21:25:47.189110 29045 raft_consensus.cc:374] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:47.189198 29045 raft_consensus.cc:729] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37e9c08ca35f419b874cf64812057324, State: Initialized, Role: FOLLOWER
I20240502 21:25:47.189325 29044 consensus_queue.cc:260] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } }
I20240502 21:25:47.189404 29045 consensus_queue.cc:260] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } }
I20240502 21:25:47.189515 29044 raft_consensus.cc:388] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:47.189545 29045 raft_consensus.cc:388] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:47.189600 29044 raft_consensus.cc:482] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:47.189610 29045 raft_consensus.cc:482] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:47.189707 29045 raft_consensus.cc:3037] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:47.189707 29044 raft_consensus.cc:3037] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:47.191244 29044 raft_consensus.cc:504] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } }
I20240502 21:25:47.191267 29045 raft_consensus.cc:504] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } }
I20240502 21:25:47.191447 29044 leader_election.cc:302] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0444722e0404faf9e70415e169e464f; no voters:
I20240502 21:25:47.191452 29045 leader_election.cc:302] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37e9c08ca35f419b874cf64812057324; no voters:
I20240502 21:25:47.191605 29045 leader_election.cc:288] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:47.191637 29044 leader_election.cc:288] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:47.191704 29000 raft_consensus.cc:2781] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:47.191720 28949 raft_consensus.cc:2781] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:47.192446 29045 ts_tablet_manager.cc:1398] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20240502 21:25:47.192955 28949 raft_consensus.cc:686] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Becoming Leader. State: Replica: e0444722e0404faf9e70415e169e464f, State: Running, Role: LEADER
I20240502 21:25:47.193418 28949 consensus_queue.cc:237] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } }
I20240502 21:25:47.194253 29000 raft_consensus.cc:686] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Becoming Leader. State: Replica: 37e9c08ca35f419b874cf64812057324, State: Running, Role: LEADER
I20240502 21:25:47.194617 29000 consensus_queue.cc:237] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } }
I20240502 21:25:47.195513 27871 catalog_manager.cc:5484] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f reported cstate change: term changed from 0 to 1, leader changed from <none> to e0444722e0404faf9e70415e169e464f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "e0444722e0404faf9e70415e169e464f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:47.202178 27871 catalog_manager.cc:5484] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37e9c08ca35f419b874cf64812057324 (127.24.226.130). New cstate: current_term: 1 leader_uuid: "37e9c08ca35f419b874cf64812057324" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:47.202649 29044 ts_tablet_manager.cc:1398] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f: Time spent starting tablet: real 0.014s user 0.002s sys 0.012s
I20240502 21:25:47.202885 29044 tablet_bootstrap.cc:492] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f: Bootstrap starting.
I20240502 21:25:47.204238 29044 tablet_bootstrap.cc:654] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:47.205935 29044 tablet_bootstrap.cc:492] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f: No bootstrap required, opened a new log
I20240502 21:25:47.206076 29044 ts_tablet_manager.cc:1367] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:25:47.206729 29044 raft_consensus.cc:348] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } }
I20240502 21:25:47.206897 29044 raft_consensus.cc:374] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:47.206971 29044 raft_consensus.cc:729] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0444722e0404faf9e70415e169e464f, State: Initialized, Role: FOLLOWER
I20240502 21:25:47.207175 29044 consensus_queue.cc:260] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } }
I20240502 21:25:47.207335 29044 raft_consensus.cc:388] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:47.207430 29044 raft_consensus.cc:482] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:47.207509 29044 raft_consensus.cc:3037] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:47.209072 29044 raft_consensus.cc:504] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } }
I20240502 21:25:47.209260 29044 leader_election.cc:302] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0444722e0404faf9e70415e169e464f; no voters:
I20240502 21:25:47.209450 29044 leader_election.cc:288] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:47.209523 29048 raft_consensus.cc:2781] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:47.209715 29048 raft_consensus.cc:686] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Becoming Leader. State: Replica: e0444722e0404faf9e70415e169e464f, State: Running, Role: LEADER
I20240502 21:25:47.209852 29044 ts_tablet_manager.cc:1398] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20240502 21:25:47.210091 29048 consensus_queue.cc:237] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } }
I20240502 21:25:47.211905 27871 catalog_manager.cc:5484] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f reported cstate change: term changed from 0 to 1, leader changed from <none> to e0444722e0404faf9e70415e169e464f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "e0444722e0404faf9e70415e169e464f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } health_report { overall_health: HEALTHY } } }
W20240502 21:25:47.500303 29055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:47.500597 29055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:47.507023 29055 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 102.199 ms
time per row: 0.0510993 ms
I20240502 21:25:47.730321 27876 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36142:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:25:47.738152 28130 tablet_service.cc:1467] Processing CreateTablet for tablet 6d31b0bb7992486f834cb67cd90eb74b (DEFAULT_TABLE table=rebalance_test_table_4 [id=3bafa5250c0e45519eb4f187c574fce7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:47.738183 28131 tablet_service.cc:1467] Processing CreateTablet for tablet 6cccd964394b451c9d602223e94acd82 (DEFAULT_TABLE table=rebalance_test_table_4 [id=3bafa5250c0e45519eb4f187c574fce7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:47.738306 28000 tablet_service.cc:1467] Processing CreateTablet for tablet 805c2e7913f34cbd883482be9583fa3c (DEFAULT_TABLE table=rebalance_test_table_4 [id=3bafa5250c0e45519eb4f187c574fce7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:47.738740 28000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 805c2e7913f34cbd883482be9583fa3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:47.738987 28130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d31b0bb7992486f834cb67cd90eb74b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:47.739272 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cccd964394b451c9d602223e94acd82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:47.744956 29098 tablet_bootstrap.cc:492] T 805c2e7913f34cbd883482be9583fa3c P e0444722e0404faf9e70415e169e464f: Bootstrap starting.
I20240502 21:25:47.745441 29097 tablet_bootstrap.cc:492] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324: Bootstrap starting.
I20240502 21:25:47.746250 29098 tablet_bootstrap.cc:654] T 805c2e7913f34cbd883482be9583fa3c P e0444722e0404faf9e70415e169e464f: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:47.746531 29097 tablet_bootstrap.cc:654] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:47.748010 29098 tablet_bootstrap.cc:492] T 805c2e7913f34cbd883482be9583fa3c P e0444722e0404faf9e70415e169e464f: No bootstrap required, opened a new log
I20240502 21:25:47.748147 29098 ts_tablet_manager.cc:1367] T 805c2e7913f34cbd883482be9583fa3c P e0444722e0404faf9e70415e169e464f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:25:47.748327 29097 tablet_bootstrap.cc:492] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324: No bootstrap required, opened a new log
I20240502 21:25:47.748471 29097 ts_tablet_manager.cc:1367] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 21:25:47.748944 29098 raft_consensus.cc:348] T 805c2e7913f34cbd883482be9583fa3c P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } }
I20240502 21:25:47.749123 29098 raft_consensus.cc:374] T 805c2e7913f34cbd883482be9583fa3c P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:47.749235 29098 raft_consensus.cc:729] T 805c2e7913f34cbd883482be9583fa3c P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0444722e0404faf9e70415e169e464f, State: Initialized, Role: FOLLOWER
I20240502 21:25:47.749197 29097 raft_consensus.cc:348] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } }
I20240502 21:25:47.749375 29097 raft_consensus.cc:374] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:47.749469 29097 raft_consensus.cc:729] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37e9c08ca35f419b874cf64812057324, State: Initialized, Role: FOLLOWER
I20240502 21:25:47.749486 29098 consensus_queue.cc:260] T 805c2e7913f34cbd883482be9583fa3c P e0444722e0404faf9e70415e169e464f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } }
I20240502 21:25:47.749670 29098 raft_consensus.cc:388] T 805c2e7913f34cbd883482be9583fa3c P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:47.749761 29098 raft_consensus.cc:482] T 805c2e7913f34cbd883482be9583fa3c P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:47.749701 29097 consensus_queue.cc:260] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } }
I20240502 21:25:47.749876 29098 raft_consensus.cc:3037] T 805c2e7913f34cbd883482be9583fa3c P e0444722e0404faf9e70415e169e464f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:47.749882 29097 raft_consensus.cc:388] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:47.749956 29097 raft_consensus.cc:482] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:47.750088 29097 raft_consensus.cc:3037] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:47.752530 29098 raft_consensus.cc:504] T 805c2e7913f34cbd883482be9583fa3c P e0444722e0404faf9e70415e169e464f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } }
I20240502 21:25:47.752676 29097 raft_consensus.cc:504] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } }
I20240502 21:25:47.752801 29098 leader_election.cc:302] T 805c2e7913f34cbd883482be9583fa3c P e0444722e0404faf9e70415e169e464f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0444722e0404faf9e70415e169e464f; no voters:
I20240502 21:25:47.752898 29097 leader_election.cc:302] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37e9c08ca35f419b874cf64812057324; no voters:
I20240502 21:25:47.752990 29098 leader_election.cc:288] T 805c2e7913f34cbd883482be9583fa3c P e0444722e0404faf9e70415e169e464f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:47.753062 29048 raft_consensus.cc:2781] T 805c2e7913f34cbd883482be9583fa3c P e0444722e0404faf9e70415e169e464f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:47.753168 29097 leader_election.cc:288] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:47.753202 29000 raft_consensus.cc:2781] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:47.754137 29098 ts_tablet_manager.cc:1398] T 805c2e7913f34cbd883482be9583fa3c P e0444722e0404faf9e70415e169e464f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 21:25:47.760932 29097 ts_tablet_manager.cc:1398] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324: Time spent starting tablet: real 0.012s user 0.004s sys 0.008s
I20240502 21:25:47.761198 29097 tablet_bootstrap.cc:492] T 6cccd964394b451c9d602223e94acd82 P 37e9c08ca35f419b874cf64812057324: Bootstrap starting.
I20240502 21:25:47.761344 29000 raft_consensus.cc:686] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Becoming Leader. State: Replica: 37e9c08ca35f419b874cf64812057324, State: Running, Role: LEADER
I20240502 21:25:47.761571 29048 raft_consensus.cc:686] T 805c2e7913f34cbd883482be9583fa3c P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Becoming Leader. State: Replica: e0444722e0404faf9e70415e169e464f, State: Running, Role: LEADER
I20240502 21:25:47.761687 29000 consensus_queue.cc:237] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } }
I20240502 21:25:47.761837 29048 consensus_queue.cc:237] T 805c2e7913f34cbd883482be9583fa3c P e0444722e0404faf9e70415e169e464f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } }
I20240502 21:25:47.762737 29097 tablet_bootstrap.cc:654] T 6cccd964394b451c9d602223e94acd82 P 37e9c08ca35f419b874cf64812057324: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:47.764767 29097 tablet_bootstrap.cc:492] T 6cccd964394b451c9d602223e94acd82 P 37e9c08ca35f419b874cf64812057324: No bootstrap required, opened a new log
I20240502 21:25:47.764946 29097 ts_tablet_manager.cc:1367] T 6cccd964394b451c9d602223e94acd82 P 37e9c08ca35f419b874cf64812057324: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240502 21:25:47.765251 27876 catalog_manager.cc:5484] T 805c2e7913f34cbd883482be9583fa3c P e0444722e0404faf9e70415e169e464f reported cstate change: term changed from 0 to 1, leader changed from <none> to e0444722e0404faf9e70415e169e464f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "e0444722e0404faf9e70415e169e464f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:47.765501 29097 raft_consensus.cc:348] T 6cccd964394b451c9d602223e94acd82 P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } }
I20240502 21:25:47.765638 29097 raft_consensus.cc:374] T 6cccd964394b451c9d602223e94acd82 P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:47.765720 29097 raft_consensus.cc:729] T 6cccd964394b451c9d602223e94acd82 P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37e9c08ca35f419b874cf64812057324, State: Initialized, Role: FOLLOWER
I20240502 21:25:47.765931 29097 consensus_queue.cc:260] T 6cccd964394b451c9d602223e94acd82 P 37e9c08ca35f419b874cf64812057324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } }
I20240502 21:25:47.766058 29097 raft_consensus.cc:388] T 6cccd964394b451c9d602223e94acd82 P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:47.766155 29097 raft_consensus.cc:482] T 6cccd964394b451c9d602223e94acd82 P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:47.766243 29097 raft_consensus.cc:3037] T 6cccd964394b451c9d602223e94acd82 P 37e9c08ca35f419b874cf64812057324 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:47.768258 29097 raft_consensus.cc:504] T 6cccd964394b451c9d602223e94acd82 P 37e9c08ca35f419b874cf64812057324 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } }
I20240502 21:25:47.768545 29097 leader_election.cc:302] T 6cccd964394b451c9d602223e94acd82 P 37e9c08ca35f419b874cf64812057324 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37e9c08ca35f419b874cf64812057324; no voters:
I20240502 21:25:47.768854 29103 raft_consensus.cc:2781] T 6cccd964394b451c9d602223e94acd82 P 37e9c08ca35f419b874cf64812057324 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:47.768739 29097 leader_election.cc:288] T 6cccd964394b451c9d602223e94acd82 P 37e9c08ca35f419b874cf64812057324 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:47.769024 29103 raft_consensus.cc:686] T 6cccd964394b451c9d602223e94acd82 P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Becoming Leader. State: Replica: 37e9c08ca35f419b874cf64812057324, State: Running, Role: LEADER
I20240502 21:25:47.769412 29103 consensus_queue.cc:237] T 6cccd964394b451c9d602223e94acd82 P 37e9c08ca35f419b874cf64812057324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } }
I20240502 21:25:47.770972 27871 catalog_manager.cc:5484] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37e9c08ca35f419b874cf64812057324 (127.24.226.130). New cstate: current_term: 1 leader_uuid: "37e9c08ca35f419b874cf64812057324" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:47.772303 29097 ts_tablet_manager.cc:1398] T 6cccd964394b451c9d602223e94acd82 P 37e9c08ca35f419b874cf64812057324: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240502 21:25:47.775171 27871 catalog_manager.cc:5484] T 6cccd964394b451c9d602223e94acd82 P 37e9c08ca35f419b874cf64812057324 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37e9c08ca35f419b874cf64812057324 (127.24.226.130). New cstate: current_term: 1 leader_uuid: "37e9c08ca35f419b874cf64812057324" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } health_report { overall_health: HEALTHY } } }
W20240502 21:25:48.072968 29110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:48.073228 29110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:48.079469 29110 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 100.295 ms
time per row: 0.0501475 ms
I20240502 21:25:48.306339 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.131:46635
--local_ip_for_outbound_sockets=127.24.226.131
--tserver_master_addrs=127.24.226.190:32867
--webserver_port=41055
--webserver_interface=127.24.226.131
--builtin_ntp_servers=127.24.226.148:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 21:25:48.594343 29143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:48.594585 29143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:48.594694 29143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:48.601883 29143 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:48.601987 29143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:48.602140 29143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.131
I20240502 21:25:48.610544 29143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.131:46635
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.24.226.131
--webserver_port=41055
--tserver_master_addrs=127.24.226.190:32867
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.131
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:48.611826 29143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:48.613461 29143 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:48.624589 29151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:48.624624 29149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:48.624827 29148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:48.625910 29143 server_base.cc:1034] running on GCE node
I20240502 21:25:48.626418 29143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:48.627303 29143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:48.628522 29143 hybrid_clock.cc:648] HybridClock initialized: now 1714685148628474 us; error 42 us; skew 500 ppm
I20240502 21:25:48.628778 29143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:48.630829 29143 webserver.cc:460] Webserver started at http://127.24.226.131:41055/ using document root <none> and password file <none>
I20240502 21:25:48.631417 29143 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:48.631522 29143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:48.635893 29143 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20240502 21:25:48.638406 29157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:48.639626 29143 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:48.639750 29143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e5b2432ac32b4c8281bfe645caf7aea2"
format_stamp: "Formatted at 2024-05-02 21:25:39 on dist-test-slave-41k1"
I20240502 21:25:48.640525 29143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:48.676656 29143 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:48.677325 29143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:48.677575 29143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:48.678436 29143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:48.679839 29143 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:48.679920 29143 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:48.680068 29143 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:48.680119 29143 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:48.708835 29143 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.131:46635
I20240502 21:25:48.708910 29269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.131:46635 every 8 connection(s)
I20240502 21:25:48.710202 29143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 21:25:48.711349 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 29143
I20240502 21:25:48.712018 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.132:38475
--local_ip_for_outbound_sockets=127.24.226.132
--tserver_master_addrs=127.24.226.190:32867
--webserver_port=38319
--webserver_interface=127.24.226.132
--builtin_ntp_servers=127.24.226.148:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:25:48.724035 29270 heartbeater.cc:347] Connected to a master server at 127.24.226.190:32867
I20240502 21:25:48.724344 29270 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:48.725395 29270 heartbeater.cc:511] Master 127.24.226.190:32867 requested a full tablet report, sending...
I20240502 21:25:48.726986 27876 ts_manager.cc:194] Re-registered known tserver with Master: e5b2432ac32b4c8281bfe645caf7aea2 (127.24.226.131:46635)
I20240502 21:25:48.727735 27876 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.131:36739
W20240502 21:25:49.006407 29274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:49.006631 29274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:49.006706 29274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:49.013134 29274 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:49.013227 29274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:49.013367 29274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.132
I20240502 21:25:49.021351 29274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.132:38475
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.24.226.132
--webserver_port=38319
--tserver_master_addrs=127.24.226.190:32867
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.132
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:49.022567 29274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:49.024142 29274 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:49.035275 29280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:49.035307 29279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:49.035351 29282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:49.036520 29274 server_base.cc:1034] running on GCE node
I20240502 21:25:49.037082 29274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:49.037941 29274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:49.039136 29274 hybrid_clock.cc:648] HybridClock initialized: now 1714685149039097 us; error 38 us; skew 500 ppm
I20240502 21:25:49.039445 29274 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:49.041635 29274 webserver.cc:460] Webserver started at http://127.24.226.132:38319/ using document root <none> and password file <none>
I20240502 21:25:49.042454 29274 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:49.042580 29274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:49.047755 29274 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20240502 21:25:49.050349 29288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:49.051609 29274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 21:25:49.051759 29274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f"
format_stamp: "Formatted at 2024-05-02 21:25:39 on dist-test-slave-41k1"
I20240502 21:25:49.052596 29274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:49.072988 29274 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:49.073674 29274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:49.073933 29274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:49.074744 29274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:49.076028 29274 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:49.076123 29274 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:49.076256 29274 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:49.076311 29274 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:49.105554 29274 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.132:38475
I20240502 21:25:49.105587 29400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.132:38475 every 8 connection(s)
I20240502 21:25:49.106890 29274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 21:25:49.110841 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 29274
I20240502 21:25:49.111449 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.133:38491
--local_ip_for_outbound_sockets=127.24.226.133
--tserver_master_addrs=127.24.226.190:32867
--webserver_port=43505
--webserver_interface=127.24.226.133
--builtin_ntp_servers=127.24.226.148:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:25:49.120685 29401 heartbeater.cc:347] Connected to a master server at 127.24.226.190:32867
I20240502 21:25:49.120975 29401 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:49.121873 29401 heartbeater.cc:511] Master 127.24.226.190:32867 requested a full tablet report, sending...
I20240502 21:25:49.123351 27876 ts_manager.cc:194] Re-registered known tserver with Master: 7b66fa9f55994d0caf5dc7d5a8189b2f (127.24.226.132:38475)
I20240502 21:25:49.124207 27876 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.132:41179
W20240502 21:25:49.403194 29404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:49.403436 29404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:49.403532 29404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:49.410288 29404 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:49.410379 29404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:49.410526 29404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.133
I20240502 21:25:49.418342 29404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.133:38491
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.24.226.133
--webserver_port=43505
--tserver_master_addrs=127.24.226.190:32867
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.133
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:49.419418 29404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:49.420882 29404 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:49.431082 29410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:49.431422 29412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:49.431663 29409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:49.431985 29404 server_base.cc:1034] running on GCE node
I20240502 21:25:49.432639 29404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:49.433643 29404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:49.434876 29404 hybrid_clock.cc:648] HybridClock initialized: now 1714685149434820 us; error 35 us; skew 500 ppm
I20240502 21:25:49.435164 29404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:49.437233 29404 webserver.cc:460] Webserver started at http://127.24.226.133:43505/ using document root <none> and password file <none>
I20240502 21:25:49.437889 29404 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:49.438009 29404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:49.442968 29404 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 21:25:49.445696 29418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:49.446920 29404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 21:25:49.447062 29404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "538c626600cd419bb5b9eeae5ded7610"
format_stamp: "Formatted at 2024-05-02 21:25:40 on dist-test-slave-41k1"
I20240502 21:25:49.447846 29404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:49.492606 29404 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:49.493335 29404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:49.493615 29404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:49.494477 29404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:49.495869 29404 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:49.495949 29404 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:49.496093 29404 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:49.496148 29404 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:49.526448 29404 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.133:38491
I20240502 21:25:49.526513 29530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.133:38491 every 8 connection(s)
I20240502 21:25:49.527725 29404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:25:49.530462 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 29404
I20240502 21:25:49.531031 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.134:37323
--local_ip_for_outbound_sockets=127.24.226.134
--tserver_master_addrs=127.24.226.190:32867
--webserver_port=39983
--webserver_interface=127.24.226.134
--builtin_ntp_servers=127.24.226.148:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:25:49.539950 29531 heartbeater.cc:347] Connected to a master server at 127.24.226.190:32867
I20240502 21:25:49.540241 29531 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:49.540940 29531 heartbeater.cc:511] Master 127.24.226.190:32867 requested a full tablet report, sending...
I20240502 21:25:49.542140 27876 ts_manager.cc:194] Re-registered known tserver with Master: 538c626600cd419bb5b9eeae5ded7610 (127.24.226.133:38491)
I20240502 21:25:49.542817 27876 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.133:33873
I20240502 21:25:49.730665 29270 heartbeater.cc:503] Master 127.24.226.190:32867 was elected leader, sending a full tablet report...
W20240502 21:25:49.824576 29534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:49.824844 29534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:49.824975 29534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:49.831432 29534 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:49.831542 29534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:49.831661 29534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.134
I20240502 21:25:49.839442 29534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.134:37323
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.24.226.134
--webserver_port=39983
--tserver_master_addrs=127.24.226.190:32867
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.134
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:49.840595 29534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:49.842011 29534 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:49.852416 29540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:49.852573 29539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:49.852592 29542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:49.853327 29534 server_base.cc:1034] running on GCE node
I20240502 21:25:49.853811 29534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:49.854601 29534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:49.855790 29534 hybrid_clock.cc:648] HybridClock initialized: now 1714685149855751 us; error 43 us; skew 500 ppm
I20240502 21:25:49.856068 29534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:49.858085 29534 webserver.cc:460] Webserver started at http://127.24.226.134:39983/ using document root <none> and password file <none>
I20240502 21:25:49.858708 29534 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:49.858827 29534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:49.863157 29534 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20240502 21:25:49.865546 29548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:49.866568 29534 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:25:49.866698 29534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6174b3a926654ec291571611f865116d"
format_stamp: "Formatted at 2024-05-02 21:25:40 on dist-test-slave-41k1"
I20240502 21:25:49.867363 29534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:49.879485 29534 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:49.880144 29534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:49.880383 29534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:49.881129 29534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:49.882294 29534 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:49.882364 29534 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:49.882479 29534 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:49.882529 29534 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:49.911193 29534 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.134:37323
I20240502 21:25:49.911227 29660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.134:37323 every 8 connection(s)
I20240502 21:25:49.912386 29534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 21:25:49.917496 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 29534
I20240502 21:25:49.918011 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.135:33083
--local_ip_for_outbound_sockets=127.24.226.135
--tserver_master_addrs=127.24.226.190:32867
--webserver_port=43919
--webserver_interface=127.24.226.135
--builtin_ntp_servers=127.24.226.148:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:25:49.923344 29661 heartbeater.cc:347] Connected to a master server at 127.24.226.190:32867
I20240502 21:25:49.923580 29661 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:49.924235 29661 heartbeater.cc:511] Master 127.24.226.190:32867 requested a full tablet report, sending...
I20240502 21:25:49.925626 27870 ts_manager.cc:194] Re-registered known tserver with Master: 6174b3a926654ec291571611f865116d (127.24.226.134:37323)
I20240502 21:25:49.926402 27870 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.134:33947
I20240502 21:25:50.126986 29401 heartbeater.cc:503] Master 127.24.226.190:32867 was elected leader, sending a full tablet report...
W20240502 21:25:50.210517 29664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:50.210772 29664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:50.210893 29664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:50.217216 29664 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:50.217344 29664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:50.217478 29664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.135
I20240502 21:25:50.224864 29664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.135:33083
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.24.226.135
--webserver_port=43919
--tserver_master_addrs=127.24.226.190:32867
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.135
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:50.225937 29664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:50.227372 29664 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:50.237308 29670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:50.237267 29669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:50.237336 29672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:50.238250 29664 server_base.cc:1034] running on GCE node
I20240502 21:25:50.238739 29664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:50.239521 29664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:50.240700 29664 hybrid_clock.cc:648] HybridClock initialized: now 1714685150240653 us; error 48 us; skew 500 ppm
I20240502 21:25:50.240953 29664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:50.242758 29664 webserver.cc:460] Webserver started at http://127.24.226.135:43919/ using document root <none> and password file <none>
I20240502 21:25:50.243314 29664 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:50.243424 29664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:50.247565 29664 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20240502 21:25:50.249953 29678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:50.251052 29664 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 21:25:50.251178 29664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "18d49d446e634bb5a630d06b99730d21"
format_stamp: "Formatted at 2024-05-02 21:25:40 on dist-test-slave-41k1"
I20240502 21:25:50.251879 29664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:50.264923 29664 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:50.265465 29664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:50.265663 29664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:50.266326 29664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:50.267458 29664 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:50.267517 29664 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:50.267588 29664 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:50.267624 29664 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:50.294967 29664 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.135:33083
I20240502 21:25:50.295024 29790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.135:33083 every 8 connection(s)
I20240502 21:25:50.296135 29664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 21:25:50.305120 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 29664
I20240502 21:25:50.306974 29791 heartbeater.cc:347] Connected to a master server at 127.24.226.190:32867
I20240502 21:25:50.307281 29791 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:50.308137 29791 heartbeater.cc:511] Master 127.24.226.190:32867 requested a full tablet report, sending...
I20240502 21:25:50.309567 27870 ts_manager.cc:194] Re-registered known tserver with Master: 18d49d446e634bb5a630d06b99730d21 (127.24.226.135:33083)
I20240502 21:25:50.310334 27870 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.135:35891
I20240502 21:25:50.544907 29531 heartbeater.cc:503] Master 127.24.226.190:32867 was elected leader, sending a full tablet report...
I20240502 21:25:50.756139 29726 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:50.760812 28130 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:25:50.774060 29466 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:50.794116 29596 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:50.802206 28000 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:25:50.807015 29336 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:50.807206 29205 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:50.928846 29661 heartbeater.cc:503] Master 127.24.226.190:32867 was elected leader, sending a full tablet report...
I20240502 21:25:50.995863 29726 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:51.007896 28130 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:25:51.016038 29466 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:51.021648 29596 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:51.054556 28000 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:25:51.055760 29336 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:51.069728 29205 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:51.133792 28020 consensus_queue.cc:237] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "538c626600cd419bb5b9eeae5ded7610" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 38491 } attrs { promote: true } }
I20240502 21:25:51.136291 29880 raft_consensus.cc:2932] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 538c626600cd419bb5b9eeae5ded7610 (127.24.226.133) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "538c626600cd419bb5b9eeae5ded7610" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 38491 } attrs { promote: true } } }
I20240502 21:25:51.141485 27875 catalog_manager.cc:5484] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f reported cstate change: config changed from index -1 to 4, NON_VOTER 538c626600cd419bb5b9eeae5ded7610 (127.24.226.133) added. New cstate: current_term: 1 leader_uuid: "e0444722e0404faf9e70415e169e464f" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "538c626600cd419bb5b9eeae5ded7610" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 38491 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 21:25:51.148739 27954 consensus_peers.cc:485] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f -> Peer 538c626600cd419bb5b9eeae5ded7610 (127.24.226.133:38491): Couldn't send request to peer 538c626600cd419bb5b9eeae5ded7610. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25affb8859f94f308ff433d9cb288de0. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:51.155913 28151 consensus_queue.cc:237] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 46635 } attrs { promote: true } }
I20240502 21:25:51.158303 29893 raft_consensus.cc:2932] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e5b2432ac32b4c8281bfe645caf7aea2 (127.24.226.131) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 46635 } attrs { promote: true } } }
I20240502 21:25:51.163074 27875 catalog_manager.cc:5484] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324 reported cstate change: config changed from index -1 to 4, NON_VOTER e5b2432ac32b4c8281bfe645caf7aea2 (127.24.226.131) added. New cstate: current_term: 1 leader_uuid: "37e9c08ca35f419b874cf64812057324" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 46635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 21:25:51.166913 28084 consensus_peers.cc:485] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324 -> Peer e5b2432ac32b4c8281bfe645caf7aea2 (127.24.226.131:46635): Couldn't send request to peer e5b2432ac32b4c8281bfe645caf7aea2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3aa757477ad450ab8d4c3029b16c670. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:51.174784 28020 consensus_queue.cc:237] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 38475 } attrs { promote: true } }
I20240502 21:25:51.176052 29879 raft_consensus.cc:2932] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7b66fa9f55994d0caf5dc7d5a8189b2f (127.24.226.132) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 38475 } attrs { promote: true } } }
I20240502 21:25:51.180928 27875 catalog_manager.cc:5484] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f reported cstate change: config changed from index -1 to 2, NON_VOTER 7b66fa9f55994d0caf5dc7d5a8189b2f (127.24.226.132) added. New cstate: current_term: 1 leader_uuid: "e0444722e0404faf9e70415e169e464f" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 38475 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 21:25:51.182973 27953 consensus_peers.cc:485] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f -> Peer 7b66fa9f55994d0caf5dc7d5a8189b2f (127.24.226.132:38475): Couldn't send request to peer 7b66fa9f55994d0caf5dc7d5a8189b2f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e18547d3903940c1ab603b8d4129b6d3. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:51.191485 28151 consensus_queue.cc:237] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "18d49d446e634bb5a630d06b99730d21" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 33083 } attrs { promote: true } }
I20240502 21:25:51.192899 29893 raft_consensus.cc:2932] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 18d49d446e634bb5a630d06b99730d21 (127.24.226.135) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "18d49d446e634bb5a630d06b99730d21" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 33083 } attrs { promote: true } } }
I20240502 21:25:51.198227 27875 catalog_manager.cc:5484] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324 reported cstate change: config changed from index -1 to 4, NON_VOTER 18d49d446e634bb5a630d06b99730d21 (127.24.226.135) added. New cstate: current_term: 1 leader_uuid: "37e9c08ca35f419b874cf64812057324" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18d49d446e634bb5a630d06b99730d21" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 33083 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 21:25:51.199604 28087 consensus_peers.cc:485] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324 -> Peer 18d49d446e634bb5a630d06b99730d21 (127.24.226.135:33083): Couldn't send request to peer 18d49d446e634bb5a630d06b99730d21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 53fea94f8310494b9354f09fc92ca06a. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:51.206764 28020 consensus_queue.cc:237] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "6174b3a926654ec291571611f865116d" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 37323 } attrs { promote: true } }
I20240502 21:25:51.208367 29879 raft_consensus.cc:2932] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6174b3a926654ec291571611f865116d (127.24.226.134) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "6174b3a926654ec291571611f865116d" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 37323 } attrs { promote: true } } }
I20240502 21:25:51.214075 27875 catalog_manager.cc:5484] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f reported cstate change: config changed from index -1 to 4, NON_VOTER 6174b3a926654ec291571611f865116d (127.24.226.134) added. New cstate: current_term: 1 leader_uuid: "e0444722e0404faf9e70415e169e464f" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6174b3a926654ec291571611f865116d" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 37323 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 21:25:51.214768 27956 consensus_peers.cc:485] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f -> Peer 6174b3a926654ec291571611f865116d (127.24.226.134:37323): Couldn't send request to peer 6174b3a926654ec291571611f865116d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a7eaa487ef4426292192a0f4768567c. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:51.222805 28151 consensus_queue.cc:237] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "538c626600cd419bb5b9eeae5ded7610" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 38491 } attrs { promote: true } }
I20240502 21:25:51.224653 29892 raft_consensus.cc:2932] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 538c626600cd419bb5b9eeae5ded7610 (127.24.226.133) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "538c626600cd419bb5b9eeae5ded7610" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 38491 } attrs { promote: true } } }
I20240502 21:25:51.229532 27875 catalog_manager.cc:5484] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324 reported cstate change: config changed from index -1 to 4, NON_VOTER 538c626600cd419bb5b9eeae5ded7610 (127.24.226.133) added. New cstate: current_term: 1 leader_uuid: "37e9c08ca35f419b874cf64812057324" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "538c626600cd419bb5b9eeae5ded7610" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 38491 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 21:25:51.229840 28085 consensus_peers.cc:485] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324 -> Peer 538c626600cd419bb5b9eeae5ded7610 (127.24.226.133:38491): Couldn't send request to peer 538c626600cd419bb5b9eeae5ded7610. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4aeae3dbc5f7474dba8ed8841feab6c3. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:51.238456 28020 consensus_queue.cc:237] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 46635 } attrs { promote: true } }
I20240502 21:25:51.239949 29879 raft_consensus.cc:2932] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e5b2432ac32b4c8281bfe645caf7aea2 (127.24.226.131) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 46635 } attrs { promote: true } } }
I20240502 21:25:51.245121 27875 catalog_manager.cc:5484] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f reported cstate change: config changed from index -1 to 4, NON_VOTER e5b2432ac32b4c8281bfe645caf7aea2 (127.24.226.131) added. New cstate: current_term: 1 leader_uuid: "e0444722e0404faf9e70415e169e464f" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 46635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 21:25:51.246785 27953 consensus_peers.cc:485] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f -> Peer e5b2432ac32b4c8281bfe645caf7aea2 (127.24.226.131:46635): Couldn't send request to peer e5b2432ac32b4c8281bfe645caf7aea2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d3b7a019c07149d9a291267a76909669. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:51.254734 28151 consensus_queue.cc:237] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 38475 } attrs { promote: true } }
I20240502 21:25:51.256124 29893 raft_consensus.cc:2932] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7b66fa9f55994d0caf5dc7d5a8189b2f (127.24.226.132) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 38475 } attrs { promote: true } } }
I20240502 21:25:51.260054 27875 catalog_manager.cc:5484] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324 reported cstate change: config changed from index -1 to 4, NON_VOTER 7b66fa9f55994d0caf5dc7d5a8189b2f (127.24.226.132) added. New cstate: current_term: 1 leader_uuid: "37e9c08ca35f419b874cf64812057324" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 38475 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 21:25:51.261811 28084 consensus_peers.cc:485] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324 -> Peer 7b66fa9f55994d0caf5dc7d5a8189b2f (127.24.226.132:38475): Couldn't send request to peer 7b66fa9f55994d0caf5dc7d5a8189b2f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c607e33cdf814d00b3d005493d941a0a. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:51.270437 28020 consensus_queue.cc:237] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "18d49d446e634bb5a630d06b99730d21" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 33083 } attrs { promote: true } }
I20240502 21:25:51.272543 29879 raft_consensus.cc:2932] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 18d49d446e634bb5a630d06b99730d21 (127.24.226.135) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "18d49d446e634bb5a630d06b99730d21" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 33083 } attrs { promote: true } } }
I20240502 21:25:51.276054 27875 catalog_manager.cc:5484] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f reported cstate change: config changed from index -1 to 4, NON_VOTER 18d49d446e634bb5a630d06b99730d21 (127.24.226.135) added. New cstate: current_term: 1 leader_uuid: "e0444722e0404faf9e70415e169e464f" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18d49d446e634bb5a630d06b99730d21" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 33083 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 21:25:51.278522 27956 consensus_peers.cc:485] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f -> Peer 18d49d446e634bb5a630d06b99730d21 (127.24.226.135:33083): Couldn't send request to peer 18d49d446e634bb5a630d06b99730d21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96f3f6d4adfc423e8c2f2ef24ba0cf1f. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:51.286074 28151 consensus_queue.cc:237] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "6174b3a926654ec291571611f865116d" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 37323 } attrs { promote: true } }
I20240502 21:25:51.287549 29892 raft_consensus.cc:2932] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6174b3a926654ec291571611f865116d (127.24.226.134) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "6174b3a926654ec291571611f865116d" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 37323 } attrs { promote: true } } }
I20240502 21:25:51.292289 27875 catalog_manager.cc:5484] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324 reported cstate change: config changed from index -1 to 4, NON_VOTER 6174b3a926654ec291571611f865116d (127.24.226.134) added. New cstate: current_term: 1 leader_uuid: "37e9c08ca35f419b874cf64812057324" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6174b3a926654ec291571611f865116d" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 37323 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 21:25:51.294338 28087 consensus_peers.cc:485] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324 -> Peer 6174b3a926654ec291571611f865116d (127.24.226.134:37323): Couldn't send request to peer 6174b3a926654ec291571611f865116d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6d31b0bb7992486f834cb67cd90eb74b. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:51.312403 29791 heartbeater.cc:503] Master 127.24.226.190:32867 was elected leader, sending a full tablet report...
I20240502 21:25:51.547650 29993 ts_tablet_manager.cc:902] T c3aa757477ad450ab8d4c3029b16c670 P e5b2432ac32b4c8281bfe645caf7aea2: Initiating tablet copy from peer 37e9c08ca35f419b874cf64812057324 (127.24.226.130:35721)
I20240502 21:25:51.549640 29993 tablet_copy_client.cc:308] T c3aa757477ad450ab8d4c3029b16c670 P e5b2432ac32b4c8281bfe645caf7aea2: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.130:35721
I20240502 21:25:51.555855 28171 tablet_copy_service.cc:140] P 37e9c08ca35f419b874cf64812057324: Received BeginTabletCopySession request for tablet c3aa757477ad450ab8d4c3029b16c670 from peer e5b2432ac32b4c8281bfe645caf7aea2 ({username='slave'} at 127.24.226.131:58411)
I20240502 21:25:51.556034 28171 tablet_copy_service.cc:161] P 37e9c08ca35f419b874cf64812057324: Beginning new tablet copy session on tablet c3aa757477ad450ab8d4c3029b16c670 from peer e5b2432ac32b4c8281bfe645caf7aea2 at {username='slave'} at 127.24.226.131:58411: session id = e5b2432ac32b4c8281bfe645caf7aea2-c3aa757477ad450ab8d4c3029b16c670
I20240502 21:25:51.558607 28171 tablet_copy_source_session.cc:205] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:25:51.562641 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3aa757477ad450ab8d4c3029b16c670. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:51.572077 29993 tablet_copy_client.cc:783] T c3aa757477ad450ab8d4c3029b16c670 P e5b2432ac32b4c8281bfe645caf7aea2: tablet copy: Starting download of 0 data blocks...
I20240502 21:25:51.572407 29993 tablet_copy_client.cc:647] T c3aa757477ad450ab8d4c3029b16c670 P e5b2432ac32b4c8281bfe645caf7aea2: tablet copy: Starting download of 1 WAL segments...
I20240502 21:25:51.574893 29993 tablet_copy_client.cc:523] T c3aa757477ad450ab8d4c3029b16c670 P e5b2432ac32b4c8281bfe645caf7aea2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:25:51.578100 29993 tablet_bootstrap.cc:492] T c3aa757477ad450ab8d4c3029b16c670 P e5b2432ac32b4c8281bfe645caf7aea2: Bootstrap starting.
I20240502 21:25:51.588936 29993 log.cc:824] T c3aa757477ad450ab8d4c3029b16c670 P e5b2432ac32b4c8281bfe645caf7aea2: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:51.630239 30025 ts_tablet_manager.cc:902] T 53fea94f8310494b9354f09fc92ca06a P 18d49d446e634bb5a630d06b99730d21: Initiating tablet copy from peer 37e9c08ca35f419b874cf64812057324 (127.24.226.130:35721)
I20240502 21:25:51.640936 30026 ts_tablet_manager.cc:902] T 25affb8859f94f308ff433d9cb288de0 P 538c626600cd419bb5b9eeae5ded7610: Initiating tablet copy from peer e0444722e0404faf9e70415e169e464f (127.24.226.129:39509)
I20240502 21:25:51.645025 29993 tablet_bootstrap.cc:492] T c3aa757477ad450ab8d4c3029b16c670 P e5b2432ac32b4c8281bfe645caf7aea2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:25:51.645517 29993 tablet_bootstrap.cc:492] T c3aa757477ad450ab8d4c3029b16c670 P e5b2432ac32b4c8281bfe645caf7aea2: Bootstrap complete.
I20240502 21:25:51.645782 30025 tablet_copy_client.cc:308] T 53fea94f8310494b9354f09fc92ca06a P 18d49d446e634bb5a630d06b99730d21: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.130:35721
I20240502 21:25:51.645877 29993 ts_tablet_manager.cc:1367] T c3aa757477ad450ab8d4c3029b16c670 P e5b2432ac32b4c8281bfe645caf7aea2: Time spent bootstrapping tablet: real 0.068s user 0.049s sys 0.006s
I20240502 21:25:51.652941 29993 raft_consensus.cc:348] T c3aa757477ad450ab8d4c3029b16c670 P e5b2432ac32b4c8281bfe645caf7aea2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 46635 } attrs { promote: true } }
I20240502 21:25:51.653478 29993 raft_consensus.cc:729] T c3aa757477ad450ab8d4c3029b16c670 P e5b2432ac32b4c8281bfe645caf7aea2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5b2432ac32b4c8281bfe645caf7aea2, State: Initialized, Role: LEARNER
I20240502 21:25:51.654147 29993 consensus_queue.cc:260] T c3aa757477ad450ab8d4c3029b16c670 P e5b2432ac32b4c8281bfe645caf7aea2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 46635 } attrs { promote: true } }
I20240502 21:25:51.656538 29993 ts_tablet_manager.cc:1398] T c3aa757477ad450ab8d4c3029b16c670 P e5b2432ac32b4c8281bfe645caf7aea2: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20240502 21:25:51.658885 28171 tablet_copy_service.cc:342] P 37e9c08ca35f419b874cf64812057324: Request end of tablet copy session e5b2432ac32b4c8281bfe645caf7aea2-c3aa757477ad450ab8d4c3029b16c670 received from {username='slave'} at 127.24.226.131:58411
I20240502 21:25:51.659229 28171 tablet_copy_service.cc:433] P 37e9c08ca35f419b874cf64812057324: ending tablet copy session e5b2432ac32b4c8281bfe645caf7aea2-c3aa757477ad450ab8d4c3029b16c670 on tablet c3aa757477ad450ab8d4c3029b16c670 with peer e5b2432ac32b4c8281bfe645caf7aea2
I20240502 21:25:51.666139 30026 tablet_copy_client.cc:308] T 25affb8859f94f308ff433d9cb288de0 P 538c626600cd419bb5b9eeae5ded7610: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.129:39509
I20240502 21:25:51.669643 28171 tablet_copy_service.cc:140] P 37e9c08ca35f419b874cf64812057324: Received BeginTabletCopySession request for tablet 53fea94f8310494b9354f09fc92ca06a from peer 18d49d446e634bb5a630d06b99730d21 ({username='slave'} at 127.24.226.135:43341)
I20240502 21:25:51.669829 28171 tablet_copy_service.cc:161] P 37e9c08ca35f419b874cf64812057324: Beginning new tablet copy session on tablet 53fea94f8310494b9354f09fc92ca06a from peer 18d49d446e634bb5a630d06b99730d21 at {username='slave'} at 127.24.226.135:43341: session id = 18d49d446e634bb5a630d06b99730d21-53fea94f8310494b9354f09fc92ca06a
I20240502 21:25:51.672619 28171 tablet_copy_source_session.cc:205] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:25:51.676357 30025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53fea94f8310494b9354f09fc92ca06a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:51.680953 30044 ts_tablet_manager.cc:902] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 538c626600cd419bb5b9eeae5ded7610: Initiating tablet copy from peer 37e9c08ca35f419b874cf64812057324 (127.24.226.130:35721)
I20240502 21:25:51.688755 30025 tablet_copy_client.cc:783] T 53fea94f8310494b9354f09fc92ca06a P 18d49d446e634bb5a630d06b99730d21: tablet copy: Starting download of 0 data blocks...
I20240502 21:25:51.689093 30025 tablet_copy_client.cc:647] T 53fea94f8310494b9354f09fc92ca06a P 18d49d446e634bb5a630d06b99730d21: tablet copy: Starting download of 1 WAL segments...
I20240502 21:25:51.689172 30046 ts_tablet_manager.cc:902] T 6d31b0bb7992486f834cb67cd90eb74b P 6174b3a926654ec291571611f865116d: Initiating tablet copy from peer 37e9c08ca35f419b874cf64812057324 (127.24.226.130:35721)
I20240502 21:25:51.691583 30025 tablet_copy_client.cc:523] T 53fea94f8310494b9354f09fc92ca06a P 18d49d446e634bb5a630d06b99730d21: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:25:51.696090 30025 tablet_bootstrap.cc:492] T 53fea94f8310494b9354f09fc92ca06a P 18d49d446e634bb5a630d06b99730d21: Bootstrap starting.
I20240502 21:25:51.696327 28040 tablet_copy_service.cc:140] P e0444722e0404faf9e70415e169e464f: Received BeginTabletCopySession request for tablet 25affb8859f94f308ff433d9cb288de0 from peer 538c626600cd419bb5b9eeae5ded7610 ({username='slave'} at 127.24.226.133:56573)
I20240502 21:25:51.696568 28040 tablet_copy_service.cc:161] P e0444722e0404faf9e70415e169e464f: Beginning new tablet copy session on tablet 25affb8859f94f308ff433d9cb288de0 from peer 538c626600cd419bb5b9eeae5ded7610 at {username='slave'} at 127.24.226.133:56573: session id = 538c626600cd419bb5b9eeae5ded7610-25affb8859f94f308ff433d9cb288de0
I20240502 21:25:51.699945 28040 tablet_copy_source_session.cc:205] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:25:51.700645 30047 ts_tablet_manager.cc:902] T e18547d3903940c1ab603b8d4129b6d3 P 7b66fa9f55994d0caf5dc7d5a8189b2f: Initiating tablet copy from peer e0444722e0404faf9e70415e169e464f (127.24.226.129:39509)
I20240502 21:25:51.704618 30026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25affb8859f94f308ff433d9cb288de0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:51.707098 30025 log.cc:824] T 53fea94f8310494b9354f09fc92ca06a P 18d49d446e634bb5a630d06b99730d21: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:51.711302 30044 tablet_copy_client.cc:308] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 538c626600cd419bb5b9eeae5ded7610: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.130:35721
I20240502 21:25:51.721844 28171 tablet_copy_service.cc:140] P 37e9c08ca35f419b874cf64812057324: Received BeginTabletCopySession request for tablet 4aeae3dbc5f7474dba8ed8841feab6c3 from peer 538c626600cd419bb5b9eeae5ded7610 ({username='slave'} at 127.24.226.133:35227)
I20240502 21:25:51.721982 28171 tablet_copy_service.cc:161] P 37e9c08ca35f419b874cf64812057324: Beginning new tablet copy session on tablet 4aeae3dbc5f7474dba8ed8841feab6c3 from peer 538c626600cd419bb5b9eeae5ded7610 at {username='slave'} at 127.24.226.133:35227: session id = 538c626600cd419bb5b9eeae5ded7610-4aeae3dbc5f7474dba8ed8841feab6c3
I20240502 21:25:51.725512 28171 tablet_copy_source_session.cc:205] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:25:51.726226 30026 tablet_copy_client.cc:783] T 25affb8859f94f308ff433d9cb288de0 P 538c626600cd419bb5b9eeae5ded7610: tablet copy: Starting download of 0 data blocks...
I20240502 21:25:51.726991 30044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aeae3dbc5f7474dba8ed8841feab6c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:51.734921 30044 tablet_copy_client.cc:783] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 538c626600cd419bb5b9eeae5ded7610: tablet copy: Starting download of 0 data blocks...
I20240502 21:25:51.735206 30044 tablet_copy_client.cc:647] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 538c626600cd419bb5b9eeae5ded7610: tablet copy: Starting download of 1 WAL segments...
I20240502 21:25:51.726969 30026 tablet_copy_client.cc:647] T 25affb8859f94f308ff433d9cb288de0 P 538c626600cd419bb5b9eeae5ded7610: tablet copy: Starting download of 1 WAL segments...
I20240502 21:25:51.737583 30044 tablet_copy_client.cc:523] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 538c626600cd419bb5b9eeae5ded7610: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:25:51.738222 30026 tablet_copy_client.cc:523] T 25affb8859f94f308ff433d9cb288de0 P 538c626600cd419bb5b9eeae5ded7610: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:25:51.740787 30047 tablet_copy_client.cc:308] T e18547d3903940c1ab603b8d4129b6d3 P 7b66fa9f55994d0caf5dc7d5a8189b2f: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.129:39509
I20240502 21:25:51.743548 30026 tablet_bootstrap.cc:492] T 25affb8859f94f308ff433d9cb288de0 P 538c626600cd419bb5b9eeae5ded7610: Bootstrap starting.
I20240502 21:25:51.744241 30064 ts_tablet_manager.cc:902] T c607e33cdf814d00b3d005493d941a0a P 7b66fa9f55994d0caf5dc7d5a8189b2f: Initiating tablet copy from peer 37e9c08ca35f419b874cf64812057324 (127.24.226.130:35721)
I20240502 21:25:51.745460 30044 tablet_bootstrap.cc:492] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 538c626600cd419bb5b9eeae5ded7610: Bootstrap starting.
I20240502 21:25:51.745944 30046 tablet_copy_client.cc:308] T 6d31b0bb7992486f834cb67cd90eb74b P 6174b3a926654ec291571611f865116d: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.130:35721
I20240502 21:25:51.763172 28040 tablet_copy_service.cc:140] P e0444722e0404faf9e70415e169e464f: Received BeginTabletCopySession request for tablet e18547d3903940c1ab603b8d4129b6d3 from peer 7b66fa9f55994d0caf5dc7d5a8189b2f ({username='slave'} at 127.24.226.132:60287)
I20240502 21:25:51.763384 28040 tablet_copy_service.cc:161] P e0444722e0404faf9e70415e169e464f: Beginning new tablet copy session on tablet e18547d3903940c1ab603b8d4129b6d3 from peer 7b66fa9f55994d0caf5dc7d5a8189b2f at {username='slave'} at 127.24.226.132:60287: session id = 7b66fa9f55994d0caf5dc7d5a8189b2f-e18547d3903940c1ab603b8d4129b6d3
I20240502 21:25:51.767067 28040 tablet_copy_source_session.cc:205] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:25:51.770391 30026 log.cc:824] T 25affb8859f94f308ff433d9cb288de0 P 538c626600cd419bb5b9eeae5ded7610: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:51.771792 30047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e18547d3903940c1ab603b8d4129b6d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:51.784118 30069 ts_tablet_manager.cc:902] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P 18d49d446e634bb5a630d06b99730d21: Initiating tablet copy from peer e0444722e0404faf9e70415e169e464f (127.24.226.129:39509)
I20240502 21:25:51.785815 30047 tablet_copy_client.cc:783] T e18547d3903940c1ab603b8d4129b6d3 P 7b66fa9f55994d0caf5dc7d5a8189b2f: tablet copy: Starting download of 0 data blocks...
I20240502 21:25:51.786126 30047 tablet_copy_client.cc:647] T e18547d3903940c1ab603b8d4129b6d3 P 7b66fa9f55994d0caf5dc7d5a8189b2f: tablet copy: Starting download of 1 WAL segments...
I20240502 21:25:51.787928 30047 tablet_copy_client.cc:523] T e18547d3903940c1ab603b8d4129b6d3 P 7b66fa9f55994d0caf5dc7d5a8189b2f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:25:51.791731 30047 tablet_bootstrap.cc:492] T e18547d3903940c1ab603b8d4129b6d3 P 7b66fa9f55994d0caf5dc7d5a8189b2f: Bootstrap starting.
I20240502 21:25:51.791916 28171 tablet_copy_service.cc:140] P 37e9c08ca35f419b874cf64812057324: Received BeginTabletCopySession request for tablet 6d31b0bb7992486f834cb67cd90eb74b from peer 6174b3a926654ec291571611f865116d ({username='slave'} at 127.24.226.134:43203)
I20240502 21:25:51.792070 28171 tablet_copy_service.cc:161] P 37e9c08ca35f419b874cf64812057324: Beginning new tablet copy session on tablet 6d31b0bb7992486f834cb67cd90eb74b from peer 6174b3a926654ec291571611f865116d at {username='slave'} at 127.24.226.134:43203: session id = 6174b3a926654ec291571611f865116d-6d31b0bb7992486f834cb67cd90eb74b
I20240502 21:25:51.795540 28171 tablet_copy_source_session.cc:205] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:25:51.799326 30047 log.cc:824] T e18547d3903940c1ab603b8d4129b6d3 P 7b66fa9f55994d0caf5dc7d5a8189b2f: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:51.799961 30046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d31b0bb7992486f834cb67cd90eb74b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:51.824674 30046 tablet_copy_client.cc:783] T 6d31b0bb7992486f834cb67cd90eb74b P 6174b3a926654ec291571611f865116d: tablet copy: Starting download of 0 data blocks...
I20240502 21:25:51.825413 30046 tablet_copy_client.cc:647] T 6d31b0bb7992486f834cb67cd90eb74b P 6174b3a926654ec291571611f865116d: tablet copy: Starting download of 1 WAL segments...
I20240502 21:25:51.828373 30046 tablet_copy_client.cc:523] T 6d31b0bb7992486f834cb67cd90eb74b P 6174b3a926654ec291571611f865116d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:25:51.834645 30078 ts_tablet_manager.cc:902] T 6a7eaa487ef4426292192a0f4768567c P 6174b3a926654ec291571611f865116d: Initiating tablet copy from peer e0444722e0404faf9e70415e169e464f (127.24.226.129:39509)
I20240502 21:25:51.835526 30047 tablet_bootstrap.cc:492] T e18547d3903940c1ab603b8d4129b6d3 P 7b66fa9f55994d0caf5dc7d5a8189b2f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:25:51.835458 30064 tablet_copy_client.cc:308] T c607e33cdf814d00b3d005493d941a0a P 7b66fa9f55994d0caf5dc7d5a8189b2f: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.130:35721
I20240502 21:25:51.835981 30047 tablet_bootstrap.cc:492] T e18547d3903940c1ab603b8d4129b6d3 P 7b66fa9f55994d0caf5dc7d5a8189b2f: Bootstrap complete.
I20240502 21:25:51.836454 30047 ts_tablet_manager.cc:1367] T e18547d3903940c1ab603b8d4129b6d3 P 7b66fa9f55994d0caf5dc7d5a8189b2f: Time spent bootstrapping tablet: real 0.045s user 0.017s sys 0.002s
I20240502 21:25:51.844213 30046 tablet_bootstrap.cc:492] T 6d31b0bb7992486f834cb67cd90eb74b P 6174b3a926654ec291571611f865116d: Bootstrap starting.
I20240502 21:25:51.854872 28171 tablet_copy_service.cc:140] P 37e9c08ca35f419b874cf64812057324: Received BeginTabletCopySession request for tablet c607e33cdf814d00b3d005493d941a0a from peer 7b66fa9f55994d0caf5dc7d5a8189b2f ({username='slave'} at 127.24.226.132:46339)
I20240502 21:25:51.855057 28171 tablet_copy_service.cc:161] P 37e9c08ca35f419b874cf64812057324: Beginning new tablet copy session on tablet c607e33cdf814d00b3d005493d941a0a from peer 7b66fa9f55994d0caf5dc7d5a8189b2f at {username='slave'} at 127.24.226.132:46339: session id = 7b66fa9f55994d0caf5dc7d5a8189b2f-c607e33cdf814d00b3d005493d941a0a
I20240502 21:25:51.855644 30069 tablet_copy_client.cc:308] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P 18d49d446e634bb5a630d06b99730d21: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.129:39509
I20240502 21:25:51.857894 28171 tablet_copy_source_session.cc:205] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:25:51.859447 30064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c607e33cdf814d00b3d005493d941a0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:51.860031 29993 ts_tablet_manager.cc:902] T d3b7a019c07149d9a291267a76909669 P e5b2432ac32b4c8281bfe645caf7aea2: Initiating tablet copy from peer e0444722e0404faf9e70415e169e464f (127.24.226.129:39509)
I20240502 21:25:51.865612 30064 tablet_copy_client.cc:783] T c607e33cdf814d00b3d005493d941a0a P 7b66fa9f55994d0caf5dc7d5a8189b2f: tablet copy: Starting download of 0 data blocks...
I20240502 21:25:51.865856 30064 tablet_copy_client.cc:647] T c607e33cdf814d00b3d005493d941a0a P 7b66fa9f55994d0caf5dc7d5a8189b2f: tablet copy: Starting download of 1 WAL segments...
I20240502 21:25:51.867056 30047 raft_consensus.cc:348] T e18547d3903940c1ab603b8d4129b6d3 P 7b66fa9f55994d0caf5dc7d5a8189b2f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 38475 } attrs { promote: true } }
I20240502 21:25:51.867511 30047 raft_consensus.cc:729] T e18547d3903940c1ab603b8d4129b6d3 P 7b66fa9f55994d0caf5dc7d5a8189b2f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7b66fa9f55994d0caf5dc7d5a8189b2f, State: Initialized, Role: LEARNER
I20240502 21:25:51.868021 30047 consensus_queue.cc:260] T e18547d3903940c1ab603b8d4129b6d3 P 7b66fa9f55994d0caf5dc7d5a8189b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 38475 } attrs { promote: true } }
I20240502 21:25:51.869144 28040 tablet_copy_service.cc:140] P e0444722e0404faf9e70415e169e464f: Received BeginTabletCopySession request for tablet 96f3f6d4adfc423e8c2f2ef24ba0cf1f from peer 18d49d446e634bb5a630d06b99730d21 ({username='slave'} at 127.24.226.135:51549)
I20240502 21:25:51.869364 28040 tablet_copy_service.cc:161] P e0444722e0404faf9e70415e169e464f: Beginning new tablet copy session on tablet 96f3f6d4adfc423e8c2f2ef24ba0cf1f from peer 18d49d446e634bb5a630d06b99730d21 at {username='slave'} at 127.24.226.135:51549: session id = 18d49d446e634bb5a630d06b99730d21-96f3f6d4adfc423e8c2f2ef24ba0cf1f
I20240502 21:25:51.873126 30078 tablet_copy_client.cc:308] T 6a7eaa487ef4426292192a0f4768567c P 6174b3a926654ec291571611f865116d: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.129:39509
I20240502 21:25:51.873157 28040 tablet_copy_source_session.cc:205] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:25:51.875149 30069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96f3f6d4adfc423e8c2f2ef24ba0cf1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:51.878378 30025 tablet_bootstrap.cc:492] T 53fea94f8310494b9354f09fc92ca06a P 18d49d446e634bb5a630d06b99730d21: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:25:51.880709 30047 ts_tablet_manager.cc:1398] T e18547d3903940c1ab603b8d4129b6d3 P 7b66fa9f55994d0caf5dc7d5a8189b2f: Time spent starting tablet: real 0.044s user 0.015s sys 0.003s
I20240502 21:25:51.880965 30064 tablet_copy_client.cc:523] T c607e33cdf814d00b3d005493d941a0a P 7b66fa9f55994d0caf5dc7d5a8189b2f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:25:51.882544 30069 tablet_copy_client.cc:783] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P 18d49d446e634bb5a630d06b99730d21: tablet copy: Starting download of 0 data blocks...
I20240502 21:25:51.882738 30069 tablet_copy_client.cc:647] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P 18d49d446e634bb5a630d06b99730d21: tablet copy: Starting download of 1 WAL segments...
I20240502 21:25:51.884047 30064 tablet_bootstrap.cc:492] T c607e33cdf814d00b3d005493d941a0a P 7b66fa9f55994d0caf5dc7d5a8189b2f: Bootstrap starting.
I20240502 21:25:51.885500 30046 log.cc:824] T 6d31b0bb7992486f834cb67cd90eb74b P 6174b3a926654ec291571611f865116d: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:51.887615 30069 tablet_copy_client.cc:523] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P 18d49d446e634bb5a630d06b99730d21: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:25:51.887619 28040 tablet_copy_service.cc:140] P e0444722e0404faf9e70415e169e464f: Received BeginTabletCopySession request for tablet 6a7eaa487ef4426292192a0f4768567c from peer 6174b3a926654ec291571611f865116d ({username='slave'} at 127.24.226.134:58759)
I20240502 21:25:51.887764 28040 tablet_copy_service.cc:161] P e0444722e0404faf9e70415e169e464f: Beginning new tablet copy session on tablet 6a7eaa487ef4426292192a0f4768567c from peer 6174b3a926654ec291571611f865116d at {username='slave'} at 127.24.226.134:58759: session id = 6174b3a926654ec291571611f865116d-6a7eaa487ef4426292192a0f4768567c
I20240502 21:25:51.890640 28040 tablet_copy_source_session.cc:205] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:25:51.891829 30078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a7eaa487ef4426292192a0f4768567c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:51.891827 28039 tablet_copy_service.cc:342] P e0444722e0404faf9e70415e169e464f: Request end of tablet copy session 7b66fa9f55994d0caf5dc7d5a8189b2f-e18547d3903940c1ab603b8d4129b6d3 received from {username='slave'} at 127.24.226.132:60287
I20240502 21:25:51.891950 28039 tablet_copy_service.cc:433] P e0444722e0404faf9e70415e169e464f: ending tablet copy session 7b66fa9f55994d0caf5dc7d5a8189b2f-e18547d3903940c1ab603b8d4129b6d3 on tablet e18547d3903940c1ab603b8d4129b6d3 with peer 7b66fa9f55994d0caf5dc7d5a8189b2f
I20240502 21:25:51.892138 30069 tablet_bootstrap.cc:492] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P 18d49d446e634bb5a630d06b99730d21: Bootstrap starting.
I20240502 21:25:51.878865 30025 tablet_bootstrap.cc:492] T 53fea94f8310494b9354f09fc92ca06a P 18d49d446e634bb5a630d06b99730d21: Bootstrap complete.
I20240502 21:25:51.898244 30025 ts_tablet_manager.cc:1367] T 53fea94f8310494b9354f09fc92ca06a P 18d49d446e634bb5a630d06b99730d21: Time spent bootstrapping tablet: real 0.202s user 0.054s sys 0.014s
I20240502 21:25:51.899392 30078 tablet_copy_client.cc:783] T 6a7eaa487ef4426292192a0f4768567c P 6174b3a926654ec291571611f865116d: tablet copy: Starting download of 0 data blocks...
I20240502 21:25:51.899718 30078 tablet_copy_client.cc:647] T 6a7eaa487ef4426292192a0f4768567c P 6174b3a926654ec291571611f865116d: tablet copy: Starting download of 1 WAL segments...
I20240502 21:25:51.901669 30078 tablet_copy_client.cc:523] T 6a7eaa487ef4426292192a0f4768567c P 6174b3a926654ec291571611f865116d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:25:51.905160 30078 tablet_bootstrap.cc:492] T 6a7eaa487ef4426292192a0f4768567c P 6174b3a926654ec291571611f865116d: Bootstrap starting.
I20240502 21:25:51.915778 30025 raft_consensus.cc:348] T 53fea94f8310494b9354f09fc92ca06a P 18d49d446e634bb5a630d06b99730d21 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "18d49d446e634bb5a630d06b99730d21" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 33083 } attrs { promote: true } }
I20240502 21:25:51.916615 30025 raft_consensus.cc:729] T 53fea94f8310494b9354f09fc92ca06a P 18d49d446e634bb5a630d06b99730d21 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 18d49d446e634bb5a630d06b99730d21, State: Initialized, Role: LEARNER
I20240502 21:25:51.917209 30025 consensus_queue.cc:260] T 53fea94f8310494b9354f09fc92ca06a P 18d49d446e634bb5a630d06b99730d21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "18d49d446e634bb5a630d06b99730d21" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 33083 } attrs { promote: true } }
I20240502 21:25:51.920639 29993 tablet_copy_client.cc:308] T d3b7a019c07149d9a291267a76909669 P e5b2432ac32b4c8281bfe645caf7aea2: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.129:39509
I20240502 21:25:51.929184 28040 tablet_copy_service.cc:140] P e0444722e0404faf9e70415e169e464f: Received BeginTabletCopySession request for tablet d3b7a019c07149d9a291267a76909669 from peer e5b2432ac32b4c8281bfe645caf7aea2 ({username='slave'} at 127.24.226.131:59447)
I20240502 21:25:51.929342 28040 tablet_copy_service.cc:161] P e0444722e0404faf9e70415e169e464f: Beginning new tablet copy session on tablet d3b7a019c07149d9a291267a76909669 from peer e5b2432ac32b4c8281bfe645caf7aea2 at {username='slave'} at 127.24.226.131:59447: session id = e5b2432ac32b4c8281bfe645caf7aea2-d3b7a019c07149d9a291267a76909669
I20240502 21:25:51.932217 28040 tablet_copy_source_session.cc:205] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:25:51.933599 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3b7a019c07149d9a291267a76909669. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:51.933851 30025 ts_tablet_manager.cc:1398] T 53fea94f8310494b9354f09fc92ca06a P 18d49d446e634bb5a630d06b99730d21: Time spent starting tablet: real 0.035s user 0.007s sys 0.003s
I20240502 21:25:51.937827 28171 tablet_copy_service.cc:342] P 37e9c08ca35f419b874cf64812057324: Request end of tablet copy session 18d49d446e634bb5a630d06b99730d21-53fea94f8310494b9354f09fc92ca06a received from {username='slave'} at 127.24.226.135:43341
I20240502 21:25:51.937987 28171 tablet_copy_service.cc:433] P 37e9c08ca35f419b874cf64812057324: ending tablet copy session 18d49d446e634bb5a630d06b99730d21-53fea94f8310494b9354f09fc92ca06a on tablet 53fea94f8310494b9354f09fc92ca06a with peer 18d49d446e634bb5a630d06b99730d21
I20240502 21:25:51.941222 29993 tablet_copy_client.cc:783] T d3b7a019c07149d9a291267a76909669 P e5b2432ac32b4c8281bfe645caf7aea2: tablet copy: Starting download of 0 data blocks...
I20240502 21:25:51.941537 29993 tablet_copy_client.cc:647] T d3b7a019c07149d9a291267a76909669 P e5b2432ac32b4c8281bfe645caf7aea2: tablet copy: Starting download of 1 WAL segments...
I20240502 21:25:51.943161 29993 tablet_copy_client.cc:523] T d3b7a019c07149d9a291267a76909669 P e5b2432ac32b4c8281bfe645caf7aea2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:25:51.948599 29225 raft_consensus.cc:1203] T c3aa757477ad450ab8d4c3029b16c670 P e5b2432ac32b4c8281bfe645caf7aea2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:25:51.949293 29993 tablet_bootstrap.cc:492] T d3b7a019c07149d9a291267a76909669 P e5b2432ac32b4c8281bfe645caf7aea2: Bootstrap starting.
I20240502 21:25:51.951164 30044 tablet_bootstrap.cc:492] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 538c626600cd419bb5b9eeae5ded7610: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:25:51.951820 30044 tablet_bootstrap.cc:492] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 538c626600cd419bb5b9eeae5ded7610: Bootstrap complete.
I20240502 21:25:51.952342 30044 ts_tablet_manager.cc:1367] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 538c626600cd419bb5b9eeae5ded7610: Time spent bootstrapping tablet: real 0.207s user 0.062s sys 0.007s
I20240502 21:25:51.958775 30044 raft_consensus.cc:348] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 538c626600cd419bb5b9eeae5ded7610 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "538c626600cd419bb5b9eeae5ded7610" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 38491 } attrs { promote: true } }
I20240502 21:25:51.959326 30044 raft_consensus.cc:729] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 538c626600cd419bb5b9eeae5ded7610 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 538c626600cd419bb5b9eeae5ded7610, State: Initialized, Role: LEARNER
I20240502 21:25:51.959933 30044 consensus_queue.cc:260] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 538c626600cd419bb5b9eeae5ded7610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "538c626600cd419bb5b9eeae5ded7610" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 38491 } attrs { promote: true } }
I20240502 21:25:51.967916 30044 ts_tablet_manager.cc:1398] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 538c626600cd419bb5b9eeae5ded7610: Time spent starting tablet: real 0.015s user 0.006s sys 0.008s
I20240502 21:25:51.968888 28171 tablet_copy_service.cc:342] P 37e9c08ca35f419b874cf64812057324: Request end of tablet copy session 538c626600cd419bb5b9eeae5ded7610-4aeae3dbc5f7474dba8ed8841feab6c3 received from {username='slave'} at 127.24.226.133:35227
I20240502 21:25:51.969043 28171 tablet_copy_service.cc:433] P 37e9c08ca35f419b874cf64812057324: ending tablet copy session 538c626600cd419bb5b9eeae5ded7610-4aeae3dbc5f7474dba8ed8841feab6c3 on tablet 4aeae3dbc5f7474dba8ed8841feab6c3 with peer 538c626600cd419bb5b9eeae5ded7610
I20240502 21:25:51.969990 30026 tablet_bootstrap.cc:492] T 25affb8859f94f308ff433d9cb288de0 P 538c626600cd419bb5b9eeae5ded7610: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:25:51.970515 30026 tablet_bootstrap.cc:492] T 25affb8859f94f308ff433d9cb288de0 P 538c626600cd419bb5b9eeae5ded7610: Bootstrap complete.
I20240502 21:25:51.970867 30026 ts_tablet_manager.cc:1367] T 25affb8859f94f308ff433d9cb288de0 P 538c626600cd419bb5b9eeae5ded7610: Time spent bootstrapping tablet: real 0.228s user 0.060s sys 0.013s
I20240502 21:25:51.971933 30026 raft_consensus.cc:348] T 25affb8859f94f308ff433d9cb288de0 P 538c626600cd419bb5b9eeae5ded7610 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "538c626600cd419bb5b9eeae5ded7610" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 38491 } attrs { promote: true } }
I20240502 21:25:51.973521 30026 raft_consensus.cc:729] T 25affb8859f94f308ff433d9cb288de0 P 538c626600cd419bb5b9eeae5ded7610 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 538c626600cd419bb5b9eeae5ded7610, State: Initialized, Role: LEARNER
I20240502 21:25:51.973803 30026 consensus_queue.cc:260] T 25affb8859f94f308ff433d9cb288de0 P 538c626600cd419bb5b9eeae5ded7610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "538c626600cd419bb5b9eeae5ded7610" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 38491 } attrs { promote: true } }
I20240502 21:25:51.974382 30026 ts_tablet_manager.cc:1398] T 25affb8859f94f308ff433d9cb288de0 P 538c626600cd419bb5b9eeae5ded7610: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:25:51.976987 28040 tablet_copy_service.cc:342] P e0444722e0404faf9e70415e169e464f: Request end of tablet copy session 538c626600cd419bb5b9eeae5ded7610-25affb8859f94f308ff433d9cb288de0 received from {username='slave'} at 127.24.226.133:56573
I20240502 21:25:51.977149 28040 tablet_copy_service.cc:433] P e0444722e0404faf9e70415e169e464f: ending tablet copy session 538c626600cd419bb5b9eeae5ded7610-25affb8859f94f308ff433d9cb288de0 on tablet 25affb8859f94f308ff433d9cb288de0 with peer 538c626600cd419bb5b9eeae5ded7610
I20240502 21:25:52.006999 30046 tablet_bootstrap.cc:492] T 6d31b0bb7992486f834cb67cd90eb74b P 6174b3a926654ec291571611f865116d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:25:52.007604 30046 tablet_bootstrap.cc:492] T 6d31b0bb7992486f834cb67cd90eb74b P 6174b3a926654ec291571611f865116d: Bootstrap complete.
I20240502 21:25:52.008116 30046 ts_tablet_manager.cc:1367] T 6d31b0bb7992486f834cb67cd90eb74b P 6174b3a926654ec291571611f865116d: Time spent bootstrapping tablet: real 0.164s user 0.054s sys 0.020s
I20240502 21:25:52.014845 30046 raft_consensus.cc:348] T 6d31b0bb7992486f834cb67cd90eb74b P 6174b3a926654ec291571611f865116d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "6174b3a926654ec291571611f865116d" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 37323 } attrs { promote: true } }
I20240502 21:25:52.015306 30046 raft_consensus.cc:729] T 6d31b0bb7992486f834cb67cd90eb74b P 6174b3a926654ec291571611f865116d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6174b3a926654ec291571611f865116d, State: Initialized, Role: LEARNER
I20240502 21:25:52.015893 30046 consensus_queue.cc:260] T 6d31b0bb7992486f834cb67cd90eb74b P 6174b3a926654ec291571611f865116d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "6174b3a926654ec291571611f865116d" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 37323 } attrs { promote: true } }
I20240502 21:25:52.016381 30069 tablet_bootstrap.cc:492] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P 18d49d446e634bb5a630d06b99730d21: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:25:52.016852 30069 tablet_bootstrap.cc:492] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P 18d49d446e634bb5a630d06b99730d21: Bootstrap complete.
I20240502 21:25:52.017172 30069 ts_tablet_manager.cc:1367] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P 18d49d446e634bb5a630d06b99730d21: Time spent bootstrapping tablet: real 0.125s user 0.052s sys 0.000s
I20240502 21:25:52.018158 30069 raft_consensus.cc:348] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P 18d49d446e634bb5a630d06b99730d21 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "18d49d446e634bb5a630d06b99730d21" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 33083 } attrs { promote: true } }
I20240502 21:25:52.018469 30069 raft_consensus.cc:729] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P 18d49d446e634bb5a630d06b99730d21 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 18d49d446e634bb5a630d06b99730d21, State: Initialized, Role: LEARNER
I20240502 21:25:52.018693 30069 consensus_queue.cc:260] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P 18d49d446e634bb5a630d06b99730d21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "18d49d446e634bb5a630d06b99730d21" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 33083 } attrs { promote: true } }
I20240502 21:25:52.021631 30069 ts_tablet_manager.cc:1398] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P 18d49d446e634bb5a630d06b99730d21: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:25:52.023841 30046 ts_tablet_manager.cc:1398] T 6d31b0bb7992486f834cb67cd90eb74b P 6174b3a926654ec291571611f865116d: Time spent starting tablet: real 0.016s user 0.010s sys 0.003s
I20240502 21:25:52.024210 28040 tablet_copy_service.cc:342] P e0444722e0404faf9e70415e169e464f: Request end of tablet copy session 18d49d446e634bb5a630d06b99730d21-96f3f6d4adfc423e8c2f2ef24ba0cf1f received from {username='slave'} at 127.24.226.135:51549
I20240502 21:25:52.024454 28040 tablet_copy_service.cc:433] P e0444722e0404faf9e70415e169e464f: ending tablet copy session 18d49d446e634bb5a630d06b99730d21-96f3f6d4adfc423e8c2f2ef24ba0cf1f on tablet 96f3f6d4adfc423e8c2f2ef24ba0cf1f with peer 18d49d446e634bb5a630d06b99730d21
I20240502 21:25:52.028712 28171 tablet_copy_service.cc:342] P 37e9c08ca35f419b874cf64812057324: Request end of tablet copy session 6174b3a926654ec291571611f865116d-6d31b0bb7992486f834cb67cd90eb74b received from {username='slave'} at 127.24.226.134:43203
I20240502 21:25:52.028856 28171 tablet_copy_service.cc:433] P 37e9c08ca35f419b874cf64812057324: ending tablet copy session 6174b3a926654ec291571611f865116d-6d31b0bb7992486f834cb67cd90eb74b on tablet 6d31b0bb7992486f834cb67cd90eb74b with peer 6174b3a926654ec291571611f865116d
I20240502 21:25:52.032768 30064 tablet_bootstrap.cc:492] T c607e33cdf814d00b3d005493d941a0a P 7b66fa9f55994d0caf5dc7d5a8189b2f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:25:52.033268 30064 tablet_bootstrap.cc:492] T c607e33cdf814d00b3d005493d941a0a P 7b66fa9f55994d0caf5dc7d5a8189b2f: Bootstrap complete.
I20240502 21:25:52.033579 30064 ts_tablet_manager.cc:1367] T c607e33cdf814d00b3d005493d941a0a P 7b66fa9f55994d0caf5dc7d5a8189b2f: Time spent bootstrapping tablet: real 0.150s user 0.052s sys 0.014s
I20240502 21:25:52.034971 30064 raft_consensus.cc:348] T c607e33cdf814d00b3d005493d941a0a P 7b66fa9f55994d0caf5dc7d5a8189b2f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 38475 } attrs { promote: true } }
I20240502 21:25:52.035236 30064 raft_consensus.cc:729] T c607e33cdf814d00b3d005493d941a0a P 7b66fa9f55994d0caf5dc7d5a8189b2f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7b66fa9f55994d0caf5dc7d5a8189b2f, State: Initialized, Role: LEARNER
I20240502 21:25:52.035449 30064 consensus_queue.cc:260] T c607e33cdf814d00b3d005493d941a0a P 7b66fa9f55994d0caf5dc7d5a8189b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 38475 } attrs { promote: true } }
I20240502 21:25:52.036108 30064 ts_tablet_manager.cc:1398] T c607e33cdf814d00b3d005493d941a0a P 7b66fa9f55994d0caf5dc7d5a8189b2f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:25:52.036774 28171 tablet_copy_service.cc:342] P 37e9c08ca35f419b874cf64812057324: Request end of tablet copy session 7b66fa9f55994d0caf5dc7d5a8189b2f-c607e33cdf814d00b3d005493d941a0a received from {username='slave'} at 127.24.226.132:46339
I20240502 21:25:52.036918 28171 tablet_copy_service.cc:433] P 37e9c08ca35f419b874cf64812057324: ending tablet copy session 7b66fa9f55994d0caf5dc7d5a8189b2f-c607e33cdf814d00b3d005493d941a0a on tablet c607e33cdf814d00b3d005493d941a0a with peer 7b66fa9f55994d0caf5dc7d5a8189b2f
I20240502 21:25:52.039304 29746 raft_consensus.cc:1203] T 53fea94f8310494b9354f09fc92ca06a P 18d49d446e634bb5a630d06b99730d21 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:25:52.039525 30078 tablet_bootstrap.cc:492] T 6a7eaa487ef4426292192a0f4768567c P 6174b3a926654ec291571611f865116d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:25:52.039935 30078 tablet_bootstrap.cc:492] T 6a7eaa487ef4426292192a0f4768567c P 6174b3a926654ec291571611f865116d: Bootstrap complete.
I20240502 21:25:52.040234 30078 ts_tablet_manager.cc:1367] T 6a7eaa487ef4426292192a0f4768567c P 6174b3a926654ec291571611f865116d: Time spent bootstrapping tablet: real 0.135s user 0.060s sys 0.012s
I20240502 21:25:52.041245 30078 raft_consensus.cc:348] T 6a7eaa487ef4426292192a0f4768567c P 6174b3a926654ec291571611f865116d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "6174b3a926654ec291571611f865116d" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 37323 } attrs { promote: true } }
I20240502 21:25:52.041477 30078 raft_consensus.cc:729] T 6a7eaa487ef4426292192a0f4768567c P 6174b3a926654ec291571611f865116d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6174b3a926654ec291571611f865116d, State: Initialized, Role: LEARNER
I20240502 21:25:52.041687 30078 consensus_queue.cc:260] T 6a7eaa487ef4426292192a0f4768567c P 6174b3a926654ec291571611f865116d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "6174b3a926654ec291571611f865116d" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 37323 } attrs { promote: true } }
I20240502 21:25:52.042416 30078 ts_tablet_manager.cc:1398] T 6a7eaa487ef4426292192a0f4768567c P 6174b3a926654ec291571611f865116d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:25:52.043576 28040 tablet_copy_service.cc:342] P e0444722e0404faf9e70415e169e464f: Request end of tablet copy session 6174b3a926654ec291571611f865116d-6a7eaa487ef4426292192a0f4768567c received from {username='slave'} at 127.24.226.134:58759
I20240502 21:25:52.043732 28040 tablet_copy_service.cc:433] P e0444722e0404faf9e70415e169e464f: ending tablet copy session 6174b3a926654ec291571611f865116d-6a7eaa487ef4426292192a0f4768567c on tablet 6a7eaa487ef4426292192a0f4768567c with peer 6174b3a926654ec291571611f865116d
I20240502 21:25:52.057525 29993 tablet_bootstrap.cc:492] T d3b7a019c07149d9a291267a76909669 P e5b2432ac32b4c8281bfe645caf7aea2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:25:52.057865 29993 tablet_bootstrap.cc:492] T d3b7a019c07149d9a291267a76909669 P e5b2432ac32b4c8281bfe645caf7aea2: Bootstrap complete.
I20240502 21:25:52.058079 29993 ts_tablet_manager.cc:1367] T d3b7a019c07149d9a291267a76909669 P e5b2432ac32b4c8281bfe645caf7aea2: Time spent bootstrapping tablet: real 0.109s user 0.034s sys 0.013s
I20240502 21:25:52.058725 29993 raft_consensus.cc:348] T d3b7a019c07149d9a291267a76909669 P e5b2432ac32b4c8281bfe645caf7aea2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 46635 } attrs { promote: true } }
I20240502 21:25:52.058887 29993 raft_consensus.cc:729] T d3b7a019c07149d9a291267a76909669 P e5b2432ac32b4c8281bfe645caf7aea2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e5b2432ac32b4c8281bfe645caf7aea2, State: Initialized, Role: LEARNER
I20240502 21:25:52.059018 29993 consensus_queue.cc:260] T d3b7a019c07149d9a291267a76909669 P e5b2432ac32b4c8281bfe645caf7aea2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 46635 } attrs { promote: true } }
I20240502 21:25:52.059460 29993 ts_tablet_manager.cc:1398] T d3b7a019c07149d9a291267a76909669 P e5b2432ac32b4c8281bfe645caf7aea2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:25:52.060148 28040 tablet_copy_service.cc:342] P e0444722e0404faf9e70415e169e464f: Request end of tablet copy session e5b2432ac32b4c8281bfe645caf7aea2-d3b7a019c07149d9a291267a76909669 received from {username='slave'} at 127.24.226.131:59447
I20240502 21:25:52.060276 28040 tablet_copy_service.cc:433] P e0444722e0404faf9e70415e169e464f: ending tablet copy session e5b2432ac32b4c8281bfe645caf7aea2-d3b7a019c07149d9a291267a76909669 on tablet d3b7a019c07149d9a291267a76909669 with peer e5b2432ac32b4c8281bfe645caf7aea2
I20240502 21:25:52.114473 29486 raft_consensus.cc:1203] T 25affb8859f94f308ff433d9cb288de0 P 538c626600cd419bb5b9eeae5ded7610 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:25:52.144121 29356 raft_consensus.cc:1203] T c607e33cdf814d00b3d005493d941a0a P 7b66fa9f55994d0caf5dc7d5a8189b2f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:25:52.173640 29356 raft_consensus.cc:1203] T e18547d3903940c1ab603b8d4129b6d3 P 7b66fa9f55994d0caf5dc7d5a8189b2f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:25:52.203356 29616 raft_consensus.cc:1203] T 6d31b0bb7992486f834cb67cd90eb74b P 6174b3a926654ec291571611f865116d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:25:52.249109 29486 raft_consensus.cc:1203] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 538c626600cd419bb5b9eeae5ded7610 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:25:52.286644 29616 raft_consensus.cc:1203] T 6a7eaa487ef4426292192a0f4768567c P 6174b3a926654ec291571611f865116d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:25:52.350185 29746 raft_consensus.cc:1203] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P 18d49d446e634bb5a630d06b99730d21 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:25:52.375042 29225 raft_consensus.cc:1203] T d3b7a019c07149d9a291267a76909669 P e5b2432ac32b4c8281bfe645caf7aea2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:25:52.499567 29892 raft_consensus.cc:1050] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324: attempting to promote NON_VOTER 18d49d446e634bb5a630d06b99730d21 to VOTER
I20240502 21:25:52.500592 29892 consensus_queue.cc:237] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "18d49d446e634bb5a630d06b99730d21" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 33083 } attrs { promote: false } }
I20240502 21:25:52.502221 29746 raft_consensus.cc:1261] T 53fea94f8310494b9354f09fc92ca06a P 18d49d446e634bb5a630d06b99730d21 [term 1 LEARNER]: Refusing update from remote peer 37e9c08ca35f419b874cf64812057324: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:25:52.502764 30111 consensus_queue.cc:1035] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18d49d446e634bb5a630d06b99730d21" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 33083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:25:52.505640 29892 raft_consensus.cc:2932] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 18d49d446e634bb5a630d06b99730d21 (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "18d49d446e634bb5a630d06b99730d21" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 33083 } attrs { promote: false } } }
I20240502 21:25:52.506278 29746 raft_consensus.cc:2932] T 53fea94f8310494b9354f09fc92ca06a P 18d49d446e634bb5a630d06b99730d21 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 18d49d446e634bb5a630d06b99730d21 (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "18d49d446e634bb5a630d06b99730d21" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 33083 } attrs { promote: false } } }
I20240502 21:25:52.510321 27875 catalog_manager.cc:5484] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324 reported cstate change: config changed from index 4 to 5, 18d49d446e634bb5a630d06b99730d21 (127.24.226.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "37e9c08ca35f419b874cf64812057324" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18d49d446e634bb5a630d06b99730d21" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 33083 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:52.534250 29892 raft_consensus.cc:1050] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324: attempting to promote NON_VOTER 7b66fa9f55994d0caf5dc7d5a8189b2f to VOTER
I20240502 21:25:52.534711 29892 consensus_queue.cc:237] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 38475 } attrs { promote: false } }
I20240502 21:25:52.536948 29892 raft_consensus.cc:1050] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324: attempting to promote NON_VOTER e5b2432ac32b4c8281bfe645caf7aea2 to VOTER
I20240502 21:25:52.536924 29356 raft_consensus.cc:1261] T c607e33cdf814d00b3d005493d941a0a P 7b66fa9f55994d0caf5dc7d5a8189b2f [term 1 LEARNER]: Refusing update from remote peer 37e9c08ca35f419b874cf64812057324: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:25:52.537385 29892 consensus_queue.cc:237] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 46635 } attrs { promote: false } }
I20240502 21:25:52.537472 30112 consensus_queue.cc:1035] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 38475 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:25:52.539199 29225 raft_consensus.cc:1261] T c3aa757477ad450ab8d4c3029b16c670 P e5b2432ac32b4c8281bfe645caf7aea2 [term 1 LEARNER]: Refusing update from remote peer 37e9c08ca35f419b874cf64812057324: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:25:52.539764 29892 consensus_queue.cc:1035] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 46635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:25:52.541304 30112 raft_consensus.cc:2932] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7b66fa9f55994d0caf5dc7d5a8189b2f (127.24.226.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 38475 } attrs { promote: false } } }
I20240502 21:25:52.542066 29356 raft_consensus.cc:2932] T c607e33cdf814d00b3d005493d941a0a P 7b66fa9f55994d0caf5dc7d5a8189b2f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7b66fa9f55994d0caf5dc7d5a8189b2f (127.24.226.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 38475 } attrs { promote: false } } }
I20240502 21:25:52.543128 30111 raft_consensus.cc:2932] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e5b2432ac32b4c8281bfe645caf7aea2 (127.24.226.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 46635 } attrs { promote: false } } }
I20240502 21:25:52.543809 29225 raft_consensus.cc:2932] T c3aa757477ad450ab8d4c3029b16c670 P e5b2432ac32b4c8281bfe645caf7aea2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e5b2432ac32b4c8281bfe645caf7aea2 (127.24.226.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 46635 } attrs { promote: false } } }
I20240502 21:25:52.547602 27873 catalog_manager.cc:5484] T c607e33cdf814d00b3d005493d941a0a P 7b66fa9f55994d0caf5dc7d5a8189b2f reported cstate change: config changed from index 4 to 5, 7b66fa9f55994d0caf5dc7d5a8189b2f (127.24.226.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "37e9c08ca35f419b874cf64812057324" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 38475 } attrs { promote: false } } }
I20240502 21:25:52.551935 27873 catalog_manager.cc:5484] T c3aa757477ad450ab8d4c3029b16c670 P e5b2432ac32b4c8281bfe645caf7aea2 reported cstate change: config changed from index 4 to 5, e5b2432ac32b4c8281bfe645caf7aea2 (127.24.226.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "37e9c08ca35f419b874cf64812057324" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 46635 } attrs { promote: false } } }
I20240502 21:25:52.599750 30118 raft_consensus.cc:1050] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f: attempting to promote NON_VOTER 7b66fa9f55994d0caf5dc7d5a8189b2f to VOTER
I20240502 21:25:52.600699 30118 consensus_queue.cc:237] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 38475 } attrs { promote: false } }
I20240502 21:25:52.602537 29356 raft_consensus.cc:1261] T e18547d3903940c1ab603b8d4129b6d3 P 7b66fa9f55994d0caf5dc7d5a8189b2f [term 1 LEARNER]: Refusing update from remote peer e0444722e0404faf9e70415e169e464f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:25:52.603050 30117 consensus_queue.cc:1035] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 38475 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:25:52.605337 30118 raft_consensus.cc:2932] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7b66fa9f55994d0caf5dc7d5a8189b2f (127.24.226.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 38475 } attrs { promote: false } } }
I20240502 21:25:52.605849 29356 raft_consensus.cc:2932] T e18547d3903940c1ab603b8d4129b6d3 P 7b66fa9f55994d0caf5dc7d5a8189b2f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7b66fa9f55994d0caf5dc7d5a8189b2f (127.24.226.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 38475 } attrs { promote: false } } }
I20240502 21:25:52.609540 27873 catalog_manager.cc:5484] T e18547d3903940c1ab603b8d4129b6d3 P 7b66fa9f55994d0caf5dc7d5a8189b2f reported cstate change: config changed from index 2 to 3, 7b66fa9f55994d0caf5dc7d5a8189b2f (127.24.226.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e0444722e0404faf9e70415e169e464f" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 38475 } attrs { promote: false } } }
I20240502 21:25:52.657653 30231 raft_consensus.cc:1050] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324: attempting to promote NON_VOTER 6174b3a926654ec291571611f865116d to VOTER
I20240502 21:25:52.658115 30231 consensus_queue.cc:237] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "6174b3a926654ec291571611f865116d" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 37323 } attrs { promote: false } }
I20240502 21:25:52.659909 29616 raft_consensus.cc:1261] T 6d31b0bb7992486f834cb67cd90eb74b P 6174b3a926654ec291571611f865116d [term 1 LEARNER]: Refusing update from remote peer 37e9c08ca35f419b874cf64812057324: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:25:52.660403 30231 consensus_queue.cc:1035] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6174b3a926654ec291571611f865116d" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 37323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:25:52.663096 30230 raft_consensus.cc:2932] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6174b3a926654ec291571611f865116d (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "6174b3a926654ec291571611f865116d" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 37323 } attrs { promote: false } } }
I20240502 21:25:52.663707 29616 raft_consensus.cc:2932] T 6d31b0bb7992486f834cb67cd90eb74b P 6174b3a926654ec291571611f865116d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6174b3a926654ec291571611f865116d (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "6174b3a926654ec291571611f865116d" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 37323 } attrs { promote: false } } }
I20240502 21:25:52.666829 27875 catalog_manager.cc:5484] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324 reported cstate change: config changed from index 4 to 5, 6174b3a926654ec291571611f865116d (127.24.226.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "37e9c08ca35f419b874cf64812057324" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6174b3a926654ec291571611f865116d" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 37323 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:52.695310 28151 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6d31b0bb7992486f834cb67cd90eb74b"
dest_uuid: "37e9c08ca35f419b874cf64812057324"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33190
I20240502 21:25:52.695508 28151 raft_consensus.cc:595] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:25:52.718688 28151 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c607e33cdf814d00b3d005493d941a0a"
dest_uuid: "37e9c08ca35f419b874cf64812057324"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33212
I20240502 21:25:52.718848 28151 raft_consensus.cc:595] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:25:52.726063 30118 raft_consensus.cc:1050] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f: attempting to promote NON_VOTER 538c626600cd419bb5b9eeae5ded7610 to VOTER
I20240502 21:25:52.726707 30118 consensus_queue.cc:237] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "538c626600cd419bb5b9eeae5ded7610" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38491 } attrs { promote: false } }
I20240502 21:25:52.727051 30231 raft_consensus.cc:1050] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324: attempting to promote NON_VOTER 538c626600cd419bb5b9eeae5ded7610 to VOTER
I20240502 21:25:52.727638 30231 consensus_queue.cc:237] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "538c626600cd419bb5b9eeae5ded7610" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38491 } attrs { promote: false } }
I20240502 21:25:52.729727 29486 raft_consensus.cc:1261] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 538c626600cd419bb5b9eeae5ded7610 [term 1 LEARNER]: Refusing update from remote peer 37e9c08ca35f419b874cf64812057324: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:25:52.730325 30231 consensus_queue.cc:1035] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324 [LEADER]: Connected to new peer: Peer: permanent_uuid: "538c626600cd419bb5b9eeae5ded7610" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38491 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:25:52.731211 29486 raft_consensus.cc:1261] T 25affb8859f94f308ff433d9cb288de0 P 538c626600cd419bb5b9eeae5ded7610 [term 1 LEARNER]: Refusing update from remote peer e0444722e0404faf9e70415e169e464f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:25:52.731894 30118 consensus_queue.cc:1035] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f [LEADER]: Connected to new peer: Peer: permanent_uuid: "538c626600cd419bb5b9eeae5ded7610" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38491 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:25:52.734699 29880 raft_consensus.cc:2932] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 538c626600cd419bb5b9eeae5ded7610 (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "538c626600cd419bb5b9eeae5ded7610" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38491 } attrs { promote: false } } }
I20240502 21:25:52.735436 30231 raft_consensus.cc:2932] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 538c626600cd419bb5b9eeae5ded7610 (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "538c626600cd419bb5b9eeae5ded7610" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38491 } attrs { promote: false } } }
I20240502 21:25:52.735414 29485 raft_consensus.cc:2932] T 25affb8859f94f308ff433d9cb288de0 P 538c626600cd419bb5b9eeae5ded7610 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 538c626600cd419bb5b9eeae5ded7610 (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "538c626600cd419bb5b9eeae5ded7610" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38491 } attrs { promote: false } } }
I20240502 21:25:52.739215 29485 raft_consensus.cc:2932] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 538c626600cd419bb5b9eeae5ded7610 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 538c626600cd419bb5b9eeae5ded7610 (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "538c626600cd419bb5b9eeae5ded7610" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38491 } attrs { promote: false } } }
I20240502 21:25:52.740463 27875 catalog_manager.cc:5484] T 25affb8859f94f308ff433d9cb288de0 P 538c626600cd419bb5b9eeae5ded7610 reported cstate change: config changed from index 4 to 5, 538c626600cd419bb5b9eeae5ded7610 (127.24.226.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e0444722e0404faf9e70415e169e464f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "538c626600cd419bb5b9eeae5ded7610" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38491 } attrs { promote: false } } }
I20240502 21:25:52.742635 27871 catalog_manager.cc:5484] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324 reported cstate change: config changed from index 4 to 5, 538c626600cd419bb5b9eeae5ded7610 (127.24.226.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "37e9c08ca35f419b874cf64812057324" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "538c626600cd419bb5b9eeae5ded7610" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38491 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:52.743237 29880 raft_consensus.cc:1050] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f: attempting to promote NON_VOTER 6174b3a926654ec291571611f865116d to VOTER
I20240502 21:25:52.743793 29880 consensus_queue.cc:237] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "6174b3a926654ec291571611f865116d" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 37323 } attrs { promote: false } }
I20240502 21:25:52.753175 29616 raft_consensus.cc:1261] T 6a7eaa487ef4426292192a0f4768567c P 6174b3a926654ec291571611f865116d [term 1 LEARNER]: Refusing update from remote peer e0444722e0404faf9e70415e169e464f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:25:52.753629 30118 consensus_queue.cc:1035] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6174b3a926654ec291571611f865116d" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 37323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:25:52.756084 29880 raft_consensus.cc:2932] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6174b3a926654ec291571611f865116d (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "6174b3a926654ec291571611f865116d" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 37323 } attrs { promote: false } } }
I20240502 21:25:52.756685 29616 raft_consensus.cc:2932] T 6a7eaa487ef4426292192a0f4768567c P 6174b3a926654ec291571611f865116d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6174b3a926654ec291571611f865116d (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "6174b3a926654ec291571611f865116d" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 37323 } attrs { promote: false } } }
I20240502 21:25:52.760622 27873 catalog_manager.cc:5484] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f reported cstate change: config changed from index 4 to 5, 6174b3a926654ec291571611f865116d (127.24.226.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e0444722e0404faf9e70415e169e464f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6174b3a926654ec291571611f865116d" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 37323 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:52.763383 28151 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c3aa757477ad450ab8d4c3029b16c670"
dest_uuid: "37e9c08ca35f419b874cf64812057324"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33236
I20240502 21:25:52.763605 28151 raft_consensus.cc:595] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:25:52.777204 28020 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "25affb8859f94f308ff433d9cb288de0"
dest_uuid: "e0444722e0404faf9e70415e169e464f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35256
I20240502 21:25:52.777395 28020 raft_consensus.cc:595] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Received request to transfer leadership
I20240502 21:25:52.790397 28020 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e18547d3903940c1ab603b8d4129b6d3"
dest_uuid: "e0444722e0404faf9e70415e169e464f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35270
I20240502 21:25:52.790551 28020 raft_consensus.cc:595] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Received request to transfer leadership
I20240502 21:25:52.803699 28151 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "53fea94f8310494b9354f09fc92ca06a"
dest_uuid: "37e9c08ca35f419b874cf64812057324"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33252
I20240502 21:25:52.803886 28151 raft_consensus.cc:595] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:25:52.806548 30117 raft_consensus.cc:1050] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f: attempting to promote NON_VOTER e5b2432ac32b4c8281bfe645caf7aea2 to VOTER
I20240502 21:25:52.807195 30117 consensus_queue.cc:237] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 46635 } attrs { promote: false } }
I20240502 21:25:52.808984 29225 raft_consensus.cc:1261] T d3b7a019c07149d9a291267a76909669 P e5b2432ac32b4c8281bfe645caf7aea2 [term 1 LEARNER]: Refusing update from remote peer e0444722e0404faf9e70415e169e464f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:25:52.809522 29880 consensus_queue.cc:1035] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 46635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:25:52.811879 30117 raft_consensus.cc:2932] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e5b2432ac32b4c8281bfe645caf7aea2 (127.24.226.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 46635 } attrs { promote: false } } }
I20240502 21:25:52.812722 29225 raft_consensus.cc:2932] T d3b7a019c07149d9a291267a76909669 P e5b2432ac32b4c8281bfe645caf7aea2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e5b2432ac32b4c8281bfe645caf7aea2 (127.24.226.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 46635 } attrs { promote: false } } }
I20240502 21:25:52.816457 27875 catalog_manager.cc:5484] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f reported cstate change: config changed from index 4 to 5, e5b2432ac32b4c8281bfe645caf7aea2 (127.24.226.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e0444722e0404faf9e70415e169e464f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 46635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:52.826184 28020 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6a7eaa487ef4426292192a0f4768567c"
dest_uuid: "e0444722e0404faf9e70415e169e464f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35284
I20240502 21:25:52.826344 28020 raft_consensus.cc:595] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Received request to transfer leadership
I20240502 21:25:52.839963 28151 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4aeae3dbc5f7474dba8ed8841feab6c3"
dest_uuid: "37e9c08ca35f419b874cf64812057324"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33264
I20240502 21:25:52.840168 28151 raft_consensus.cc:595] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:25:52.856304 30117 raft_consensus.cc:1050] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f: attempting to promote NON_VOTER 18d49d446e634bb5a630d06b99730d21 to VOTER
I20240502 21:25:52.856791 30117 consensus_queue.cc:237] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "18d49d446e634bb5a630d06b99730d21" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 33083 } attrs { promote: false } }
I20240502 21:25:52.858311 29746 raft_consensus.cc:1261] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P 18d49d446e634bb5a630d06b99730d21 [term 1 LEARNER]: Refusing update from remote peer e0444722e0404faf9e70415e169e464f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:25:52.858737 29880 consensus_queue.cc:1035] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f [LEADER]: Connected to new peer: Peer: permanent_uuid: "18d49d446e634bb5a630d06b99730d21" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 33083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:25:52.860837 30117 raft_consensus.cc:2932] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 18d49d446e634bb5a630d06b99730d21 (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "18d49d446e634bb5a630d06b99730d21" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 33083 } attrs { promote: false } } }
I20240502 21:25:52.861373 29746 raft_consensus.cc:2932] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P 18d49d446e634bb5a630d06b99730d21 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 18d49d446e634bb5a630d06b99730d21 (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "18d49d446e634bb5a630d06b99730d21" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 33083 } attrs { promote: false } } }
I20240502 21:25:52.864797 27873 catalog_manager.cc:5484] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f reported cstate change: config changed from index 4 to 5, 18d49d446e634bb5a630d06b99730d21 (127.24.226.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e0444722e0404faf9e70415e169e464f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18d49d446e634bb5a630d06b99730d21" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 33083 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:52.886998 29892 raft_consensus.cc:979] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324: : Instructing follower 18d49d446e634bb5a630d06b99730d21 to start an election
I20240502 21:25:52.887182 30231 raft_consensus.cc:1067] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Signalling peer 18d49d446e634bb5a630d06b99730d21 to start an election
I20240502 21:25:52.887917 29746 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "53fea94f8310494b9354f09fc92ca06a"
dest_uuid: "18d49d446e634bb5a630d06b99730d21"
from {username='slave'} at 127.24.226.130:48523
I20240502 21:25:52.888144 29746 raft_consensus.cc:482] T 53fea94f8310494b9354f09fc92ca06a P 18d49d446e634bb5a630d06b99730d21 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:25:52.888324 29746 raft_consensus.cc:3037] T 53fea94f8310494b9354f09fc92ca06a P 18d49d446e634bb5a630d06b99730d21 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:52.890237 29746 raft_consensus.cc:504] T 53fea94f8310494b9354f09fc92ca06a P 18d49d446e634bb5a630d06b99730d21 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "18d49d446e634bb5a630d06b99730d21" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 33083 } attrs { promote: false } }
I20240502 21:25:52.891364 29746 leader_election.cc:288] T 53fea94f8310494b9354f09fc92ca06a P 18d49d446e634bb5a630d06b99730d21 [CANDIDATE]: Term 2 election: Requested vote from peers 37e9c08ca35f419b874cf64812057324 (127.24.226.130:35721)
I20240502 21:25:52.891827 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fea94f8310494b9354f09fc92ca06a" candidate_uuid: "18d49d446e634bb5a630d06b99730d21" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "37e9c08ca35f419b874cf64812057324"
I20240502 21:25:52.892079 28151 raft_consensus.cc:3032] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:25:52.892153 28151 raft_consensus.cc:729] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 37e9c08ca35f419b874cf64812057324, State: Running, Role: LEADER
I20240502 21:25:52.892405 28151 consensus_queue.cc:260] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "18d49d446e634bb5a630d06b99730d21" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 33083 } attrs { promote: false } }
I20240502 21:25:52.892678 28151 raft_consensus.cc:3037] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:52.894510 28151 raft_consensus.cc:2454] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18d49d446e634bb5a630d06b99730d21 in term 2.
I20240502 21:25:52.894989 29682 leader_election.cc:302] T 53fea94f8310494b9354f09fc92ca06a P 18d49d446e634bb5a630d06b99730d21 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 18d49d446e634bb5a630d06b99730d21, 37e9c08ca35f419b874cf64812057324; no voters:
I20240502 21:25:52.895305 30109 raft_consensus.cc:2781] T 53fea94f8310494b9354f09fc92ca06a P 18d49d446e634bb5a630d06b99730d21 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:25:52.895673 30109 raft_consensus.cc:686] T 53fea94f8310494b9354f09fc92ca06a P 18d49d446e634bb5a630d06b99730d21 [term 2 LEADER]: Becoming Leader. State: Replica: 18d49d446e634bb5a630d06b99730d21, State: Running, Role: LEADER
I20240502 21:25:52.896121 30109 consensus_queue.cc:237] T 53fea94f8310494b9354f09fc92ca06a P 18d49d446e634bb5a630d06b99730d21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "18d49d446e634bb5a630d06b99730d21" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 33083 } attrs { promote: false } }
I20240502 21:25:52.898929 27875 catalog_manager.cc:5484] T 53fea94f8310494b9354f09fc92ca06a P 18d49d446e634bb5a630d06b99730d21 reported cstate change: term changed from 1 to 2, leader changed from 37e9c08ca35f419b874cf64812057324 (127.24.226.130) to 18d49d446e634bb5a630d06b99730d21 (127.24.226.135). New cstate: current_term: 2 leader_uuid: "18d49d446e634bb5a630d06b99730d21" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18d49d446e634bb5a630d06b99730d21" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 33083 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 21:25:52.908394 27865 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 53fea94f8310494b9354f09fc92ca06a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 18d49d446e634bb5a630d06b99730d21 (127.24.226.135:33083) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240502 21:25:52.920349 29892 raft_consensus.cc:979] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324: : Instructing follower e5b2432ac32b4c8281bfe645caf7aea2 to start an election
I20240502 21:25:52.920538 30231 raft_consensus.cc:1067] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Signalling peer e5b2432ac32b4c8281bfe645caf7aea2 to start an election
I20240502 21:25:52.921137 29225 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c3aa757477ad450ab8d4c3029b16c670"
dest_uuid: "e5b2432ac32b4c8281bfe645caf7aea2"
from {username='slave'} at 127.24.226.130:35503
I20240502 21:25:52.921375 29225 raft_consensus.cc:482] T c3aa757477ad450ab8d4c3029b16c670 P e5b2432ac32b4c8281bfe645caf7aea2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:25:52.921571 29225 raft_consensus.cc:3037] T c3aa757477ad450ab8d4c3029b16c670 P e5b2432ac32b4c8281bfe645caf7aea2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:52.923436 29225 raft_consensus.cc:504] T c3aa757477ad450ab8d4c3029b16c670 P e5b2432ac32b4c8281bfe645caf7aea2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 46635 } attrs { promote: false } }
I20240502 21:25:52.924547 29225 leader_election.cc:288] T c3aa757477ad450ab8d4c3029b16c670 P e5b2432ac32b4c8281bfe645caf7aea2 [CANDIDATE]: Term 2 election: Requested vote from peers 37e9c08ca35f419b874cf64812057324 (127.24.226.130:35721)
I20240502 21:25:52.924952 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3aa757477ad450ab8d4c3029b16c670" candidate_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "37e9c08ca35f419b874cf64812057324"
I20240502 21:25:52.925123 28151 raft_consensus.cc:3032] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:25:52.925199 28151 raft_consensus.cc:729] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 37e9c08ca35f419b874cf64812057324, State: Running, Role: LEADER
I20240502 21:25:52.925410 28151 consensus_queue.cc:260] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 46635 } attrs { promote: false } }
I20240502 21:25:52.925643 28151 raft_consensus.cc:3037] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:52.927325 28151 raft_consensus.cc:2454] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5b2432ac32b4c8281bfe645caf7aea2 in term 2.
I20240502 21:25:52.927775 29161 leader_election.cc:302] T c3aa757477ad450ab8d4c3029b16c670 P e5b2432ac32b4c8281bfe645caf7aea2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 37e9c08ca35f419b874cf64812057324, e5b2432ac32b4c8281bfe645caf7aea2; no voters:
I20240502 21:25:52.928205 30037 raft_consensus.cc:2781] T c3aa757477ad450ab8d4c3029b16c670 P e5b2432ac32b4c8281bfe645caf7aea2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:25:52.928675 30037 raft_consensus.cc:686] T c3aa757477ad450ab8d4c3029b16c670 P e5b2432ac32b4c8281bfe645caf7aea2 [term 2 LEADER]: Becoming Leader. State: Replica: e5b2432ac32b4c8281bfe645caf7aea2, State: Running, Role: LEADER
I20240502 21:25:52.929123 30037 consensus_queue.cc:237] T c3aa757477ad450ab8d4c3029b16c670 P e5b2432ac32b4c8281bfe645caf7aea2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 46635 } attrs { promote: false } }
I20240502 21:25:52.931887 27875 catalog_manager.cc:5484] T c3aa757477ad450ab8d4c3029b16c670 P e5b2432ac32b4c8281bfe645caf7aea2 reported cstate change: term changed from 1 to 2, leader changed from 37e9c08ca35f419b874cf64812057324 (127.24.226.130) to e5b2432ac32b4c8281bfe645caf7aea2 (127.24.226.131). New cstate: current_term: 2 leader_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 46635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:53.000151 30118 raft_consensus.cc:979] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f: : Instructing follower 7b66fa9f55994d0caf5dc7d5a8189b2f to start an election
I20240502 21:25:53.000380 30117 raft_consensus.cc:1067] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Signalling peer 7b66fa9f55994d0caf5dc7d5a8189b2f to start an election
I20240502 21:25:53.001140 29356 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e18547d3903940c1ab603b8d4129b6d3"
dest_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f"
from {username='slave'} at 127.24.226.129:54399
I20240502 21:25:53.001358 29356 raft_consensus.cc:482] T e18547d3903940c1ab603b8d4129b6d3 P 7b66fa9f55994d0caf5dc7d5a8189b2f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:25:53.001552 29356 raft_consensus.cc:3037] T e18547d3903940c1ab603b8d4129b6d3 P 7b66fa9f55994d0caf5dc7d5a8189b2f [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:53.003378 29356 raft_consensus.cc:504] T e18547d3903940c1ab603b8d4129b6d3 P 7b66fa9f55994d0caf5dc7d5a8189b2f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 38475 } attrs { promote: false } }
I20240502 21:25:53.004556 29356 leader_election.cc:288] T e18547d3903940c1ab603b8d4129b6d3 P 7b66fa9f55994d0caf5dc7d5a8189b2f [CANDIDATE]: Term 2 election: Requested vote from peers e0444722e0404faf9e70415e169e464f (127.24.226.129:39509)
I20240502 21:25:53.005067 28020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e18547d3903940c1ab603b8d4129b6d3" candidate_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e0444722e0404faf9e70415e169e464f"
I20240502 21:25:53.005301 28020 raft_consensus.cc:3032] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:25:53.005374 28020 raft_consensus.cc:729] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e0444722e0404faf9e70415e169e464f, State: Running, Role: LEADER
I20240502 21:25:53.005635 28020 consensus_queue.cc:260] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 38475 } attrs { promote: false } }
I20240502 21:25:53.005921 28020 raft_consensus.cc:3037] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:53.007843 28020 raft_consensus.cc:2454] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b66fa9f55994d0caf5dc7d5a8189b2f in term 2.
I20240502 21:25:53.008322 29290 leader_election.cc:302] T e18547d3903940c1ab603b8d4129b6d3 P 7b66fa9f55994d0caf5dc7d5a8189b2f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7b66fa9f55994d0caf5dc7d5a8189b2f, e0444722e0404faf9e70415e169e464f; no voters:
I20240502 21:25:53.008656 30099 raft_consensus.cc:2781] T e18547d3903940c1ab603b8d4129b6d3 P 7b66fa9f55994d0caf5dc7d5a8189b2f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:25:53.009100 30099 raft_consensus.cc:686] T e18547d3903940c1ab603b8d4129b6d3 P 7b66fa9f55994d0caf5dc7d5a8189b2f [term 2 LEADER]: Becoming Leader. State: Replica: 7b66fa9f55994d0caf5dc7d5a8189b2f, State: Running, Role: LEADER
I20240502 21:25:53.009600 30099 consensus_queue.cc:237] T e18547d3903940c1ab603b8d4129b6d3 P 7b66fa9f55994d0caf5dc7d5a8189b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 38475 } attrs { promote: false } }
I20240502 21:25:53.012277 27875 catalog_manager.cc:5484] T e18547d3903940c1ab603b8d4129b6d3 P 7b66fa9f55994d0caf5dc7d5a8189b2f reported cstate change: term changed from 1 to 2, leader changed from e0444722e0404faf9e70415e169e464f (127.24.226.129) to 7b66fa9f55994d0caf5dc7d5a8189b2f (127.24.226.132). New cstate: current_term: 2 leader_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 38475 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:53.061895 28151 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6d31b0bb7992486f834cb67cd90eb74b"
dest_uuid: "37e9c08ca35f419b874cf64812057324"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33288
I20240502 21:25:53.062124 28151 raft_consensus.cc:595] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:25:53.076735 28020 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "96f3f6d4adfc423e8c2f2ef24ba0cf1f"
dest_uuid: "e0444722e0404faf9e70415e169e464f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35300
I20240502 21:25:53.076925 28020 raft_consensus.cc:595] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Received request to transfer leadership
I20240502 21:25:53.091023 28151 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c607e33cdf814d00b3d005493d941a0a"
dest_uuid: "37e9c08ca35f419b874cf64812057324"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33298
I20240502 21:25:53.091212 28151 raft_consensus.cc:595] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:25:53.105573 28020 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d3b7a019c07149d9a291267a76909669"
dest_uuid: "e0444722e0404faf9e70415e169e464f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35322
I20240502 21:25:53.105767 28020 raft_consensus.cc:595] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Received request to transfer leadership
I20240502 21:25:53.119361 30231 raft_consensus.cc:979] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324: : Instructing follower 7b66fa9f55994d0caf5dc7d5a8189b2f to start an election
I20240502 21:25:53.119532 30231 raft_consensus.cc:1067] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Signalling peer 7b66fa9f55994d0caf5dc7d5a8189b2f to start an election
I20240502 21:25:53.120168 29356 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c607e33cdf814d00b3d005493d941a0a"
dest_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f"
from {username='slave'} at 127.24.226.130:54879
I20240502 21:25:53.120477 29356 raft_consensus.cc:482] T c607e33cdf814d00b3d005493d941a0a P 7b66fa9f55994d0caf5dc7d5a8189b2f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:25:53.120611 29356 raft_consensus.cc:3037] T c607e33cdf814d00b3d005493d941a0a P 7b66fa9f55994d0caf5dc7d5a8189b2f [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:53.124015 29356 raft_consensus.cc:504] T c607e33cdf814d00b3d005493d941a0a P 7b66fa9f55994d0caf5dc7d5a8189b2f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 38475 } attrs { promote: false } }
I20240502 21:25:53.124619 29356 leader_election.cc:288] T c607e33cdf814d00b3d005493d941a0a P 7b66fa9f55994d0caf5dc7d5a8189b2f [CANDIDATE]: Term 2 election: Requested vote from peers 37e9c08ca35f419b874cf64812057324 (127.24.226.130:35721)
I20240502 21:25:53.125101 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c607e33cdf814d00b3d005493d941a0a" candidate_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "37e9c08ca35f419b874cf64812057324"
I20240502 21:25:53.125275 28151 raft_consensus.cc:3032] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:25:53.125341 28151 raft_consensus.cc:729] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 37e9c08ca35f419b874cf64812057324, State: Running, Role: LEADER
I20240502 21:25:53.125592 28151 consensus_queue.cc:260] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 38475 } attrs { promote: false } }
I20240502 21:25:53.125849 28151 raft_consensus.cc:3037] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:53.128093 28151 raft_consensus.cc:2454] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b66fa9f55994d0caf5dc7d5a8189b2f in term 2.
I20240502 21:25:53.128551 29292 leader_election.cc:302] T c607e33cdf814d00b3d005493d941a0a P 7b66fa9f55994d0caf5dc7d5a8189b2f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 37e9c08ca35f419b874cf64812057324, 7b66fa9f55994d0caf5dc7d5a8189b2f; no voters:
I20240502 21:25:53.128815 30099 raft_consensus.cc:2781] T c607e33cdf814d00b3d005493d941a0a P 7b66fa9f55994d0caf5dc7d5a8189b2f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:25:53.129020 30099 raft_consensus.cc:686] T c607e33cdf814d00b3d005493d941a0a P 7b66fa9f55994d0caf5dc7d5a8189b2f [term 2 LEADER]: Becoming Leader. State: Replica: 7b66fa9f55994d0caf5dc7d5a8189b2f, State: Running, Role: LEADER
I20240502 21:25:53.129290 30099 consensus_queue.cc:237] T c607e33cdf814d00b3d005493d941a0a P 7b66fa9f55994d0caf5dc7d5a8189b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 38475 } attrs { promote: false } }
I20240502 21:25:53.131502 27875 catalog_manager.cc:5484] T c607e33cdf814d00b3d005493d941a0a P 7b66fa9f55994d0caf5dc7d5a8189b2f reported cstate change: term changed from 1 to 2, leader changed from 37e9c08ca35f419b874cf64812057324 (127.24.226.130) to 7b66fa9f55994d0caf5dc7d5a8189b2f (127.24.226.132). New cstate: current_term: 2 leader_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 38475 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:53.138046 28020 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "25affb8859f94f308ff433d9cb288de0"
dest_uuid: "e0444722e0404faf9e70415e169e464f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35342
I20240502 21:25:53.138236 28020 raft_consensus.cc:595] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Received request to transfer leadership
I20240502 21:25:53.170840 28020 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6a7eaa487ef4426292192a0f4768567c"
dest_uuid: "e0444722e0404faf9e70415e169e464f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35360
I20240502 21:25:53.171052 28020 raft_consensus.cc:595] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Received request to transfer leadership
I20240502 21:25:53.186230 28151 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4aeae3dbc5f7474dba8ed8841feab6c3"
dest_uuid: "37e9c08ca35f419b874cf64812057324"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33308
I20240502 21:25:53.186398 28151 raft_consensus.cc:595] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:25:53.258285 30118 raft_consensus.cc:979] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f: : Instructing follower 6174b3a926654ec291571611f865116d to start an election
I20240502 21:25:53.258450 30118 raft_consensus.cc:1067] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Signalling peer 6174b3a926654ec291571611f865116d to start an election
I20240502 21:25:53.259090 29616 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6a7eaa487ef4426292192a0f4768567c"
dest_uuid: "6174b3a926654ec291571611f865116d"
from {username='slave'} at 127.24.226.129:55139
I20240502 21:25:53.259385 29616 raft_consensus.cc:482] T 6a7eaa487ef4426292192a0f4768567c P 6174b3a926654ec291571611f865116d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:25:53.259578 29616 raft_consensus.cc:3037] T 6a7eaa487ef4426292192a0f4768567c P 6174b3a926654ec291571611f865116d [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:53.261716 29616 raft_consensus.cc:504] T 6a7eaa487ef4426292192a0f4768567c P 6174b3a926654ec291571611f865116d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "6174b3a926654ec291571611f865116d" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 37323 } attrs { promote: false } }
I20240502 21:25:53.263020 29616 leader_election.cc:288] T 6a7eaa487ef4426292192a0f4768567c P 6174b3a926654ec291571611f865116d [CANDIDATE]: Term 2 election: Requested vote from peers e0444722e0404faf9e70415e169e464f (127.24.226.129:39509)
I20240502 21:25:53.263561 28020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a7eaa487ef4426292192a0f4768567c" candidate_uuid: "6174b3a926654ec291571611f865116d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e0444722e0404faf9e70415e169e464f"
I20240502 21:25:53.263753 28020 raft_consensus.cc:3032] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:25:53.263823 28020 raft_consensus.cc:729] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e0444722e0404faf9e70415e169e464f, State: Running, Role: LEADER
I20240502 21:25:53.264045 28020 consensus_queue.cc:260] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "6174b3a926654ec291571611f865116d" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 37323 } attrs { promote: false } }
I20240502 21:25:53.264279 28020 raft_consensus.cc:3037] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:53.266259 28020 raft_consensus.cc:2454] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6174b3a926654ec291571611f865116d in term 2.
I20240502 21:25:53.266706 29550 leader_election.cc:302] T 6a7eaa487ef4426292192a0f4768567c P 6174b3a926654ec291571611f865116d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6174b3a926654ec291571611f865116d, e0444722e0404faf9e70415e169e464f; no voters:
I20240502 21:25:53.267139 30230 raft_consensus.cc:979] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324: : Instructing follower 538c626600cd419bb5b9eeae5ded7610 to start an election
I20240502 21:25:53.267314 30230 raft_consensus.cc:1067] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Signalling peer 538c626600cd419bb5b9eeae5ded7610 to start an election
I20240502 21:25:53.267478 30428 raft_consensus.cc:2781] T 6a7eaa487ef4426292192a0f4768567c P 6174b3a926654ec291571611f865116d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:25:53.267963 30428 raft_consensus.cc:686] T 6a7eaa487ef4426292192a0f4768567c P 6174b3a926654ec291571611f865116d [term 2 LEADER]: Becoming Leader. State: Replica: 6174b3a926654ec291571611f865116d, State: Running, Role: LEADER
I20240502 21:25:53.268064 29485 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4aeae3dbc5f7474dba8ed8841feab6c3"
dest_uuid: "538c626600cd419bb5b9eeae5ded7610"
from {username='slave'} at 127.24.226.130:46915
I20240502 21:25:53.268337 29485 raft_consensus.cc:482] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 538c626600cd419bb5b9eeae5ded7610 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:25:53.268528 29485 raft_consensus.cc:3037] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 538c626600cd419bb5b9eeae5ded7610 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:53.268545 30428 consensus_queue.cc:237] T 6a7eaa487ef4426292192a0f4768567c P 6174b3a926654ec291571611f865116d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "6174b3a926654ec291571611f865116d" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 37323 } attrs { promote: false } }
I20240502 21:25:53.271687 27875 catalog_manager.cc:5484] T 6a7eaa487ef4426292192a0f4768567c P 6174b3a926654ec291571611f865116d reported cstate change: term changed from 1 to 2, leader changed from e0444722e0404faf9e70415e169e464f (127.24.226.129) to 6174b3a926654ec291571611f865116d (127.24.226.134). New cstate: current_term: 2 leader_uuid: "6174b3a926654ec291571611f865116d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6174b3a926654ec291571611f865116d" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 37323 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:53.272256 29485 raft_consensus.cc:504] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 538c626600cd419bb5b9eeae5ded7610 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "538c626600cd419bb5b9eeae5ded7610" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38491 } attrs { promote: false } }
I20240502 21:25:53.273888 29485 leader_election.cc:288] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 538c626600cd419bb5b9eeae5ded7610 [CANDIDATE]: Term 2 election: Requested vote from peers 37e9c08ca35f419b874cf64812057324 (127.24.226.130:35721)
I20240502 21:25:53.274323 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aeae3dbc5f7474dba8ed8841feab6c3" candidate_uuid: "538c626600cd419bb5b9eeae5ded7610" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "37e9c08ca35f419b874cf64812057324"
I20240502 21:25:53.274607 28151 raft_consensus.cc:3032] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:25:53.274711 28151 raft_consensus.cc:729] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 37e9c08ca35f419b874cf64812057324, State: Running, Role: LEADER
I20240502 21:25:53.275003 28151 consensus_queue.cc:260] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "538c626600cd419bb5b9eeae5ded7610" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38491 } attrs { promote: false } }
I20240502 21:25:53.275265 28151 raft_consensus.cc:3037] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:53.277776 28151 raft_consensus.cc:2454] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 538c626600cd419bb5b9eeae5ded7610 in term 2.
I20240502 21:25:53.278323 29422 leader_election.cc:302] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 538c626600cd419bb5b9eeae5ded7610 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 37e9c08ca35f419b874cf64812057324, 538c626600cd419bb5b9eeae5ded7610; no voters:
I20240502 21:25:53.279126 30431 raft_consensus.cc:2781] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 538c626600cd419bb5b9eeae5ded7610 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:25:53.279708 30431 raft_consensus.cc:686] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 538c626600cd419bb5b9eeae5ded7610 [term 2 LEADER]: Becoming Leader. State: Replica: 538c626600cd419bb5b9eeae5ded7610, State: Running, Role: LEADER
I20240502 21:25:53.280248 30431 consensus_queue.cc:237] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 538c626600cd419bb5b9eeae5ded7610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "538c626600cd419bb5b9eeae5ded7610" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38491 } attrs { promote: false } }
I20240502 21:25:53.283567 27875 catalog_manager.cc:5484] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 538c626600cd419bb5b9eeae5ded7610 reported cstate change: term changed from 1 to 2, leader changed from 37e9c08ca35f419b874cf64812057324 (127.24.226.130) to 538c626600cd419bb5b9eeae5ded7610 (127.24.226.133). New cstate: current_term: 2 leader_uuid: "538c626600cd419bb5b9eeae5ded7610" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "538c626600cd419bb5b9eeae5ded7610" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38491 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:53.285851 29892 raft_consensus.cc:979] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324: : Instructing follower 6174b3a926654ec291571611f865116d to start an election
I20240502 21:25:53.285979 29892 raft_consensus.cc:1067] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Signalling peer 6174b3a926654ec291571611f865116d to start an election
I20240502 21:25:53.287036 29616 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6d31b0bb7992486f834cb67cd90eb74b"
dest_uuid: "6174b3a926654ec291571611f865116d"
from {username='slave'} at 127.24.226.130:52803
I20240502 21:25:53.287287 29616 raft_consensus.cc:482] T 6d31b0bb7992486f834cb67cd90eb74b P 6174b3a926654ec291571611f865116d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:25:53.287412 29616 raft_consensus.cc:3037] T 6d31b0bb7992486f834cb67cd90eb74b P 6174b3a926654ec291571611f865116d [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:53.290840 29616 raft_consensus.cc:504] T 6d31b0bb7992486f834cb67cd90eb74b P 6174b3a926654ec291571611f865116d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "6174b3a926654ec291571611f865116d" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 37323 } attrs { promote: false } }
I20240502 21:25:53.292065 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d31b0bb7992486f834cb67cd90eb74b" candidate_uuid: "6174b3a926654ec291571611f865116d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "37e9c08ca35f419b874cf64812057324"
I20240502 21:25:53.292310 28151 raft_consensus.cc:3032] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:25:53.292405 28151 raft_consensus.cc:729] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 37e9c08ca35f419b874cf64812057324, State: Running, Role: LEADER
I20240502 21:25:53.292555 29616 leader_election.cc:288] T 6d31b0bb7992486f834cb67cd90eb74b P 6174b3a926654ec291571611f865116d [CANDIDATE]: Term 2 election: Requested vote from peers 37e9c08ca35f419b874cf64812057324 (127.24.226.130:35721)
I20240502 21:25:53.292753 28151 consensus_queue.cc:260] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "6174b3a926654ec291571611f865116d" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 37323 } attrs { promote: false } }
I20240502 21:25:53.293112 28151 raft_consensus.cc:3037] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:53.296312 28151 raft_consensus.cc:2454] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6174b3a926654ec291571611f865116d in term 2.
I20240502 21:25:53.296756 29552 leader_election.cc:302] T 6d31b0bb7992486f834cb67cd90eb74b P 6174b3a926654ec291571611f865116d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 37e9c08ca35f419b874cf64812057324, 6174b3a926654ec291571611f865116d; no voters:
I20240502 21:25:53.297047 30428 raft_consensus.cc:2781] T 6d31b0bb7992486f834cb67cd90eb74b P 6174b3a926654ec291571611f865116d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:25:53.297264 30428 raft_consensus.cc:686] T 6d31b0bb7992486f834cb67cd90eb74b P 6174b3a926654ec291571611f865116d [term 2 LEADER]: Becoming Leader. State: Replica: 6174b3a926654ec291571611f865116d, State: Running, Role: LEADER
I20240502 21:25:53.297503 30428 consensus_queue.cc:237] T 6d31b0bb7992486f834cb67cd90eb74b P 6174b3a926654ec291571611f865116d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "6174b3a926654ec291571611f865116d" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 37323 } attrs { promote: false } }
I20240502 21:25:53.299289 27875 catalog_manager.cc:5484] T 6d31b0bb7992486f834cb67cd90eb74b P 6174b3a926654ec291571611f865116d reported cstate change: term changed from 1 to 2, leader changed from 37e9c08ca35f419b874cf64812057324 (127.24.226.130) to 6174b3a926654ec291571611f865116d (127.24.226.134). New cstate: current_term: 2 leader_uuid: "6174b3a926654ec291571611f865116d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6174b3a926654ec291571611f865116d" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 37323 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:53.303368 30117 raft_consensus.cc:979] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f: : Instructing follower 18d49d446e634bb5a630d06b99730d21 to start an election
I20240502 21:25:53.303560 30117 raft_consensus.cc:1067] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Signalling peer 18d49d446e634bb5a630d06b99730d21 to start an election
I20240502 21:25:53.304276 29746 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "96f3f6d4adfc423e8c2f2ef24ba0cf1f"
dest_uuid: "18d49d446e634bb5a630d06b99730d21"
from {username='slave'} at 127.24.226.129:39399
I20240502 21:25:53.304499 29746 raft_consensus.cc:482] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P 18d49d446e634bb5a630d06b99730d21 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:25:53.304607 29746 raft_consensus.cc:3037] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P 18d49d446e634bb5a630d06b99730d21 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:53.306855 29746 raft_consensus.cc:504] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P 18d49d446e634bb5a630d06b99730d21 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "18d49d446e634bb5a630d06b99730d21" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 33083 } attrs { promote: false } }
I20240502 21:25:53.307699 30118 raft_consensus.cc:979] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f: : Instructing follower 538c626600cd419bb5b9eeae5ded7610 to start an election
I20240502 21:25:53.307849 30118 raft_consensus.cc:1067] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Signalling peer 538c626600cd419bb5b9eeae5ded7610 to start an election
I20240502 21:25:53.307991 29746 leader_election.cc:288] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P 18d49d446e634bb5a630d06b99730d21 [CANDIDATE]: Term 2 election: Requested vote from peers e0444722e0404faf9e70415e169e464f (127.24.226.129:39509)
I20240502 21:25:53.308475 29485 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "25affb8859f94f308ff433d9cb288de0"
dest_uuid: "538c626600cd419bb5b9eeae5ded7610"
from {username='slave'} at 127.24.226.129:48443
I20240502 21:25:53.308571 28020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f3f6d4adfc423e8c2f2ef24ba0cf1f" candidate_uuid: "18d49d446e634bb5a630d06b99730d21" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e0444722e0404faf9e70415e169e464f"
I20240502 21:25:53.308699 29485 raft_consensus.cc:482] T 25affb8859f94f308ff433d9cb288de0 P 538c626600cd419bb5b9eeae5ded7610 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:25:53.308773 29485 raft_consensus.cc:3037] T 25affb8859f94f308ff433d9cb288de0 P 538c626600cd419bb5b9eeae5ded7610 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:53.308818 28020 raft_consensus.cc:3032] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:25:53.308943 28020 raft_consensus.cc:729] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e0444722e0404faf9e70415e169e464f, State: Running, Role: LEADER
I20240502 21:25:53.309283 28020 consensus_queue.cc:260] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "18d49d446e634bb5a630d06b99730d21" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 33083 } attrs { promote: false } }
I20240502 21:25:53.309629 28020 raft_consensus.cc:3037] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:53.310629 29485 raft_consensus.cc:504] T 25affb8859f94f308ff433d9cb288de0 P 538c626600cd419bb5b9eeae5ded7610 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "538c626600cd419bb5b9eeae5ded7610" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38491 } attrs { promote: false } }
I20240502 21:25:53.311064 29485 leader_election.cc:288] T 25affb8859f94f308ff433d9cb288de0 P 538c626600cd419bb5b9eeae5ded7610 [CANDIDATE]: Term 2 election: Requested vote from peers e0444722e0404faf9e70415e169e464f (127.24.226.129:39509)
I20240502 21:25:53.311651 28019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25affb8859f94f308ff433d9cb288de0" candidate_uuid: "538c626600cd419bb5b9eeae5ded7610" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e0444722e0404faf9e70415e169e464f"
I20240502 21:25:53.311897 28019 raft_consensus.cc:3032] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:25:53.311972 28019 raft_consensus.cc:729] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e0444722e0404faf9e70415e169e464f, State: Running, Role: LEADER
I20240502 21:25:53.312198 28019 consensus_queue.cc:260] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "538c626600cd419bb5b9eeae5ded7610" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38491 } attrs { promote: false } }
I20240502 21:25:53.312461 28019 raft_consensus.cc:3037] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:53.313062 28020 raft_consensus.cc:2454] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18d49d446e634bb5a630d06b99730d21 in term 2.
I20240502 21:25:53.313525 29680 leader_election.cc:302] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P 18d49d446e634bb5a630d06b99730d21 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 18d49d446e634bb5a630d06b99730d21, e0444722e0404faf9e70415e169e464f; no voters:
I20240502 21:25:53.314023 30109 raft_consensus.cc:2781] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P 18d49d446e634bb5a630d06b99730d21 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:25:53.314206 30109 raft_consensus.cc:686] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P 18d49d446e634bb5a630d06b99730d21 [term 2 LEADER]: Becoming Leader. State: Replica: 18d49d446e634bb5a630d06b99730d21, State: Running, Role: LEADER
I20240502 21:25:53.314479 30109 consensus_queue.cc:237] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P 18d49d446e634bb5a630d06b99730d21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "18d49d446e634bb5a630d06b99730d21" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 33083 } attrs { promote: false } }
I20240502 21:25:53.314705 28019 raft_consensus.cc:2454] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 538c626600cd419bb5b9eeae5ded7610 in term 2.
I20240502 21:25:53.315069 29420 leader_election.cc:302] T 25affb8859f94f308ff433d9cb288de0 P 538c626600cd419bb5b9eeae5ded7610 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 538c626600cd419bb5b9eeae5ded7610, e0444722e0404faf9e70415e169e464f; no voters:
I20240502 21:25:53.316372 30431 raft_consensus.cc:2781] T 25affb8859f94f308ff433d9cb288de0 P 538c626600cd419bb5b9eeae5ded7610 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:25:53.316589 30431 raft_consensus.cc:686] T 25affb8859f94f308ff433d9cb288de0 P 538c626600cd419bb5b9eeae5ded7610 [term 2 LEADER]: Becoming Leader. State: Replica: 538c626600cd419bb5b9eeae5ded7610, State: Running, Role: LEADER
I20240502 21:25:53.316843 30431 consensus_queue.cc:237] T 25affb8859f94f308ff433d9cb288de0 P 538c626600cd419bb5b9eeae5ded7610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "538c626600cd419bb5b9eeae5ded7610" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38491 } attrs { promote: false } }
I20240502 21:25:53.317248 27875 catalog_manager.cc:5484] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P 18d49d446e634bb5a630d06b99730d21 reported cstate change: term changed from 1 to 2, leader changed from e0444722e0404faf9e70415e169e464f (127.24.226.129) to 18d49d446e634bb5a630d06b99730d21 (127.24.226.135). New cstate: current_term: 2 leader_uuid: "18d49d446e634bb5a630d06b99730d21" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18d49d446e634bb5a630d06b99730d21" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 33083 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:53.319685 27873 catalog_manager.cc:5484] T 25affb8859f94f308ff433d9cb288de0 P 538c626600cd419bb5b9eeae5ded7610 reported cstate change: term changed from 1 to 2, leader changed from e0444722e0404faf9e70415e169e464f (127.24.226.129) to 538c626600cd419bb5b9eeae5ded7610 (127.24.226.133). New cstate: current_term: 2 leader_uuid: "538c626600cd419bb5b9eeae5ded7610" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "538c626600cd419bb5b9eeae5ded7610" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38491 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:53.383826 29880 raft_consensus.cc:979] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f: : Instructing follower e5b2432ac32b4c8281bfe645caf7aea2 to start an election
I20240502 21:25:53.384089 29880 raft_consensus.cc:1067] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Signalling peer e5b2432ac32b4c8281bfe645caf7aea2 to start an election
I20240502 21:25:53.384826 29225 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3b7a019c07149d9a291267a76909669"
dest_uuid: "e5b2432ac32b4c8281bfe645caf7aea2"
from {username='slave'} at 127.24.226.129:44605
I20240502 21:25:53.385043 29225 raft_consensus.cc:482] T d3b7a019c07149d9a291267a76909669 P e5b2432ac32b4c8281bfe645caf7aea2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:25:53.385190 29225 raft_consensus.cc:3037] T d3b7a019c07149d9a291267a76909669 P e5b2432ac32b4c8281bfe645caf7aea2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:53.387300 29225 raft_consensus.cc:504] T d3b7a019c07149d9a291267a76909669 P e5b2432ac32b4c8281bfe645caf7aea2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 46635 } attrs { promote: false } }
I20240502 21:25:53.387931 29225 leader_election.cc:288] T d3b7a019c07149d9a291267a76909669 P e5b2432ac32b4c8281bfe645caf7aea2 [CANDIDATE]: Term 2 election: Requested vote from peers e0444722e0404faf9e70415e169e464f (127.24.226.129:39509)
I20240502 21:25:53.388643 28019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b7a019c07149d9a291267a76909669" candidate_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e0444722e0404faf9e70415e169e464f"
I20240502 21:25:53.388935 28019 raft_consensus.cc:3032] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:25:53.389055 28019 raft_consensus.cc:729] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e0444722e0404faf9e70415e169e464f, State: Running, Role: LEADER
I20240502 21:25:53.389374 28019 consensus_queue.cc:260] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 46635 } attrs { promote: false } }
I20240502 21:25:53.389757 28019 raft_consensus.cc:3037] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:53.394444 28019 raft_consensus.cc:2454] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5b2432ac32b4c8281bfe645caf7aea2 in term 2.
I20240502 21:25:53.394908 29159 leader_election.cc:302] T d3b7a019c07149d9a291267a76909669 P e5b2432ac32b4c8281bfe645caf7aea2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: e0444722e0404faf9e70415e169e464f, e5b2432ac32b4c8281bfe645caf7aea2; no voters:
I20240502 21:25:53.395223 30037 raft_consensus.cc:2781] T d3b7a019c07149d9a291267a76909669 P e5b2432ac32b4c8281bfe645caf7aea2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:25:53.395390 30037 raft_consensus.cc:686] T d3b7a019c07149d9a291267a76909669 P e5b2432ac32b4c8281bfe645caf7aea2 [term 2 LEADER]: Becoming Leader. State: Replica: e5b2432ac32b4c8281bfe645caf7aea2, State: Running, Role: LEADER
I20240502 21:25:53.395432 28151 raft_consensus.cc:1261] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324 [term 2 FOLLOWER]: Refusing update from remote peer 18d49d446e634bb5a630d06b99730d21: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:25:53.395654 30037 consensus_queue.cc:237] T d3b7a019c07149d9a291267a76909669 P e5b2432ac32b4c8281bfe645caf7aea2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 46635 } attrs { promote: false } }
I20240502 21:25:53.396277 30336 consensus_queue.cc:1035] T 53fea94f8310494b9354f09fc92ca06a P 18d49d446e634bb5a630d06b99730d21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:25:53.397835 27873 catalog_manager.cc:5484] T d3b7a019c07149d9a291267a76909669 P e5b2432ac32b4c8281bfe645caf7aea2 reported cstate change: term changed from 1 to 2, leader changed from e0444722e0404faf9e70415e169e464f (127.24.226.129) to e5b2432ac32b4c8281bfe645caf7aea2 (127.24.226.131). New cstate: current_term: 2 leader_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 46635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:53.428895 28151 raft_consensus.cc:1261] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324 [term 2 FOLLOWER]: Refusing update from remote peer e5b2432ac32b4c8281bfe645caf7aea2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:25:53.429560 30339 consensus_queue.cc:1035] T c3aa757477ad450ab8d4c3029b16c670 P e5b2432ac32b4c8281bfe645caf7aea2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:25:53.474817 28019 raft_consensus.cc:1261] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f [term 2 FOLLOWER]: Refusing update from remote peer 7b66fa9f55994d0caf5dc7d5a8189b2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:25:53.475631 30341 consensus_queue.cc:1035] T e18547d3903940c1ab603b8d4129b6d3 P 7b66fa9f55994d0caf5dc7d5a8189b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:25:53.560956 29746 consensus_queue.cc:237] T 53fea94f8310494b9354f09fc92ca06a P 18d49d446e634bb5a630d06b99730d21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "18d49d446e634bb5a630d06b99730d21" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 33083 } attrs { promote: false } }
I20240502 21:25:53.561658 30336 raft_consensus.cc:2932] T 53fea94f8310494b9354f09fc92ca06a P 18d49d446e634bb5a630d06b99730d21 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 37e9c08ca35f419b874cf64812057324 (127.24.226.130) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "18d49d446e634bb5a630d06b99730d21" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 33083 } attrs { promote: false } } }
I20240502 21:25:53.564091 27865 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 53fea94f8310494b9354f09fc92ca06a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 21:25:53.565181 27873 catalog_manager.cc:5484] T 53fea94f8310494b9354f09fc92ca06a P 18d49d446e634bb5a630d06b99730d21 reported cstate change: config changed from index 5 to 7, VOTER 37e9c08ca35f419b874cf64812057324 (127.24.226.130) evicted. New cstate: current_term: 2 leader_uuid: "18d49d446e634bb5a630d06b99730d21" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "18d49d446e634bb5a630d06b99730d21" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 33083 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:53.568194 28130 tablet_service.cc:1514] Processing DeleteTablet for tablet 53fea94f8310494b9354f09fc92ca06a with delete_type TABLET_DATA_TOMBSTONED (TS 37e9c08ca35f419b874cf64812057324 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49646
I20240502 21:25:53.569039 30489 tablet_replica.cc:330] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324: stopping tablet replica
I20240502 21:25:53.569417 30489 raft_consensus.cc:2229] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:25:53.569703 30489 raft_consensus.cc:2258] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:53.570973 30489 ts_tablet_manager.cc:1875] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:25:53.575587 30489 ts_tablet_manager.cc:1888] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 21:25:53.575755 30489 log.cc:1196] T 53fea94f8310494b9354f09fc92ca06a P 37e9c08ca35f419b874cf64812057324: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/53fea94f8310494b9354f09fc92ca06a
I20240502 21:25:53.576612 27865 catalog_manager.cc:4830] TS 37e9c08ca35f419b874cf64812057324 (127.24.226.130:35721): tablet 53fea94f8310494b9354f09fc92ca06a (table rebalance_test_table_0 [id=affadeda486b4ca9b4db96ff7f6c53cc]) successfully deleted
I20240502 21:25:53.576673 29225 consensus_queue.cc:237] T c3aa757477ad450ab8d4c3029b16c670 P e5b2432ac32b4c8281bfe645caf7aea2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 46635 } attrs { promote: false } }
I20240502 21:25:53.577430 30339 raft_consensus.cc:2932] T c3aa757477ad450ab8d4c3029b16c670 P e5b2432ac32b4c8281bfe645caf7aea2 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 37e9c08ca35f419b874cf64812057324 (127.24.226.130) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 46635 } attrs { promote: false } } }
I20240502 21:25:53.579913 27862 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c3aa757477ad450ab8d4c3029b16c670 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 21:25:53.581055 27873 catalog_manager.cc:5484] T c3aa757477ad450ab8d4c3029b16c670 P e5b2432ac32b4c8281bfe645caf7aea2 reported cstate change: config changed from index 5 to 7, VOTER 37e9c08ca35f419b874cf64812057324 (127.24.226.130) evicted. New cstate: current_term: 2 leader_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 46635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:53.583797 28130 tablet_service.cc:1514] Processing DeleteTablet for tablet c3aa757477ad450ab8d4c3029b16c670 with delete_type TABLET_DATA_TOMBSTONED (TS 37e9c08ca35f419b874cf64812057324 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49646
I20240502 21:25:53.584160 30489 tablet_replica.cc:330] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324: stopping tablet replica
I20240502 21:25:53.584353 30489 raft_consensus.cc:2229] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:25:53.584658 30489 raft_consensus.cc:2258] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:53.585388 30489 ts_tablet_manager.cc:1875] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:25:53.589287 30489 ts_tablet_manager.cc:1888] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 21:25:53.589408 30489 log.cc:1196] T c3aa757477ad450ab8d4c3029b16c670 P 37e9c08ca35f419b874cf64812057324: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/c3aa757477ad450ab8d4c3029b16c670
I20240502 21:25:53.590071 27865 catalog_manager.cc:4830] TS 37e9c08ca35f419b874cf64812057324 (127.24.226.130:35721): tablet c3aa757477ad450ab8d4c3029b16c670 (table rebalance_test_table_0 [id=affadeda486b4ca9b4db96ff7f6c53cc]) successfully deleted
I20240502 21:25:53.627067 28151 raft_consensus.cc:1261] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324 [term 2 FOLLOWER]: Refusing update from remote peer 7b66fa9f55994d0caf5dc7d5a8189b2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:25:53.627566 30341 consensus_queue.cc:1035] T c607e33cdf814d00b3d005493d941a0a P 7b66fa9f55994d0caf5dc7d5a8189b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:25:53.675717 29356 consensus_queue.cc:237] T e18547d3903940c1ab603b8d4129b6d3 P 7b66fa9f55994d0caf5dc7d5a8189b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 38475 } attrs { promote: false } }
I20240502 21:25:53.676406 30341 raft_consensus.cc:2932] T e18547d3903940c1ab603b8d4129b6d3 P 7b66fa9f55994d0caf5dc7d5a8189b2f [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER e0444722e0404faf9e70415e169e464f (127.24.226.129) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 38475 } attrs { promote: false } } }
I20240502 21:25:53.678988 27862 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e18547d3903940c1ab603b8d4129b6d3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 21:25:53.680214 27875 catalog_manager.cc:5484] T e18547d3903940c1ab603b8d4129b6d3 P 7b66fa9f55994d0caf5dc7d5a8189b2f reported cstate change: config changed from index 3 to 5, VOTER e0444722e0404faf9e70415e169e464f (127.24.226.129) evicted. New cstate: current_term: 2 leader_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 38475 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:53.683162 28000 tablet_service.cc:1514] Processing DeleteTablet for tablet e18547d3903940c1ab603b8d4129b6d3 with delete_type TABLET_DATA_TOMBSTONED (TS e0444722e0404faf9e70415e169e464f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44274
I20240502 21:25:53.683924 30491 tablet_replica.cc:330] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f: stopping tablet replica
I20240502 21:25:53.684314 30491 raft_consensus.cc:2229] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:25:53.684609 30491 raft_consensus.cc:2258] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:53.685739 30491 ts_tablet_manager.cc:1875] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:25:53.690411 30491 ts_tablet_manager.cc:1888] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 21:25:53.690531 30491 log.cc:1196] T e18547d3903940c1ab603b8d4129b6d3 P e0444722e0404faf9e70415e169e464f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/e18547d3903940c1ab603b8d4129b6d3
I20240502 21:25:53.691246 27863 catalog_manager.cc:4830] TS e0444722e0404faf9e70415e169e464f (127.24.226.129:39509): tablet e18547d3903940c1ab603b8d4129b6d3 (table TestTable [id=c0e550d8c6444707a5179f3ec075781e]) successfully deleted
I20240502 21:25:53.705502 28019 raft_consensus.cc:1261] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f [term 2 FOLLOWER]: Refusing update from remote peer 18d49d446e634bb5a630d06b99730d21: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:25:53.706049 30109 consensus_queue.cc:1035] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P 18d49d446e634bb5a630d06b99730d21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:25:53.726613 28151 raft_consensus.cc:1261] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324 [term 2 FOLLOWER]: Refusing update from remote peer 538c626600cd419bb5b9eeae5ded7610: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:25:53.727216 30431 consensus_queue.cc:1035] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 538c626600cd419bb5b9eeae5ded7610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:25:53.731659 29356 consensus_queue.cc:237] T c607e33cdf814d00b3d005493d941a0a P 7b66fa9f55994d0caf5dc7d5a8189b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 38475 } attrs { promote: false } }
I20240502 21:25:53.732410 30341 raft_consensus.cc:2932] T c607e33cdf814d00b3d005493d941a0a P 7b66fa9f55994d0caf5dc7d5a8189b2f [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 37e9c08ca35f419b874cf64812057324 (127.24.226.130) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 38475 } attrs { promote: false } } }
I20240502 21:25:53.735394 27862 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c607e33cdf814d00b3d005493d941a0a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 21:25:53.736701 27875 catalog_manager.cc:5484] T c607e33cdf814d00b3d005493d941a0a P 7b66fa9f55994d0caf5dc7d5a8189b2f reported cstate change: config changed from index 5 to 7, VOTER 37e9c08ca35f419b874cf64812057324 (127.24.226.130) evicted. New cstate: current_term: 2 leader_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7b66fa9f55994d0caf5dc7d5a8189b2f" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 38475 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:53.739737 28130 tablet_service.cc:1514] Processing DeleteTablet for tablet c607e33cdf814d00b3d005493d941a0a with delete_type TABLET_DATA_TOMBSTONED (TS 37e9c08ca35f419b874cf64812057324 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49646
I20240502 21:25:53.740118 30489 tablet_replica.cc:330] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324: stopping tablet replica
I20240502 21:25:53.740410 30489 raft_consensus.cc:2229] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:25:53.740774 30489 raft_consensus.cc:2258] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:53.741914 30489 ts_tablet_manager.cc:1875] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:25:53.748479 30489 ts_tablet_manager.cc:1888] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 21:25:53.748641 30489 log.cc:1196] T c607e33cdf814d00b3d005493d941a0a P 37e9c08ca35f419b874cf64812057324: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/c607e33cdf814d00b3d005493d941a0a
I20240502 21:25:53.749843 27865 catalog_manager.cc:4830] TS 37e9c08ca35f419b874cf64812057324 (127.24.226.130:35721): tablet c607e33cdf814d00b3d005493d941a0a (table rebalance_test_table_2 [id=6fc5508a149a46b6b19e75ddf5817ae1]) successfully deleted
I20240502 21:25:53.750672 28019 raft_consensus.cc:1261] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f [term 2 FOLLOWER]: Refusing update from remote peer 538c626600cd419bb5b9eeae5ded7610: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:25:53.751441 30432 consensus_queue.cc:1035] T 25affb8859f94f308ff433d9cb288de0 P 538c626600cd419bb5b9eeae5ded7610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:25:53.760535 28151 raft_consensus.cc:1261] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324 [term 2 FOLLOWER]: Refusing update from remote peer 6174b3a926654ec291571611f865116d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:25:53.761234 30428 consensus_queue.cc:1035] T 6d31b0bb7992486f834cb67cd90eb74b P 6174b3a926654ec291571611f865116d [LEADER]: Connected to new peer: Peer: permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:25:53.842818 28019 raft_consensus.cc:1261] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f [term 2 FOLLOWER]: Refusing update from remote peer 6174b3a926654ec291571611f865116d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:25:53.843439 30428 consensus_queue.cc:1035] T 6a7eaa487ef4426292192a0f4768567c P 6174b3a926654ec291571611f865116d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:25:53.902686 28019 raft_consensus.cc:1261] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f [term 2 FOLLOWER]: Refusing update from remote peer e5b2432ac32b4c8281bfe645caf7aea2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:25:53.903183 30339 consensus_queue.cc:1035] T d3b7a019c07149d9a291267a76909669 P e5b2432ac32b4c8281bfe645caf7aea2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:25:53.905274 29485 consensus_queue.cc:237] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 538c626600cd419bb5b9eeae5ded7610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "538c626600cd419bb5b9eeae5ded7610" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38491 } attrs { promote: false } }
I20240502 21:25:53.906050 30431 raft_consensus.cc:2932] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 538c626600cd419bb5b9eeae5ded7610 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 37e9c08ca35f419b874cf64812057324 (127.24.226.130) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "538c626600cd419bb5b9eeae5ded7610" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38491 } attrs { promote: false } } }
I20240502 21:25:53.909382 27863 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4aeae3dbc5f7474dba8ed8841feab6c3 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 21:25:53.910280 27875 catalog_manager.cc:5484] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 538c626600cd419bb5b9eeae5ded7610 reported cstate change: config changed from index 5 to 7, VOTER 37e9c08ca35f419b874cf64812057324 (127.24.226.130) evicted. New cstate: current_term: 2 leader_uuid: "538c626600cd419bb5b9eeae5ded7610" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "538c626600cd419bb5b9eeae5ded7610" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38491 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:53.913338 28130 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aeae3dbc5f7474dba8ed8841feab6c3 with delete_type TABLET_DATA_TOMBSTONED (TS 37e9c08ca35f419b874cf64812057324 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49646
I20240502 21:25:53.913654 30489 tablet_replica.cc:330] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324: stopping tablet replica
I20240502 21:25:53.913825 30489 raft_consensus.cc:2229] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:25:53.914005 30489 raft_consensus.cc:2258] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:53.914736 30489 ts_tablet_manager.cc:1875] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:25:53.919245 30489 ts_tablet_manager.cc:1888] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 21:25:53.919431 30489 log.cc:1196] T 4aeae3dbc5f7474dba8ed8841feab6c3 P 37e9c08ca35f419b874cf64812057324: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/4aeae3dbc5f7474dba8ed8841feab6c3
I20240502 21:25:53.920070 27865 catalog_manager.cc:4830] TS 37e9c08ca35f419b874cf64812057324 (127.24.226.130:35721): tablet 4aeae3dbc5f7474dba8ed8841feab6c3 (table rebalance_test_table_1 [id=f3b24b58d96b4bceb7b44fc6fe502ad0]) successfully deleted
I20240502 21:25:53.926833 29485 consensus_queue.cc:237] T 25affb8859f94f308ff433d9cb288de0 P 538c626600cd419bb5b9eeae5ded7610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "538c626600cd419bb5b9eeae5ded7610" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38491 } attrs { promote: false } }
I20240502 21:25:53.927464 30431 raft_consensus.cc:2932] T 25affb8859f94f308ff433d9cb288de0 P 538c626600cd419bb5b9eeae5ded7610 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e0444722e0404faf9e70415e169e464f (127.24.226.129) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "538c626600cd419bb5b9eeae5ded7610" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38491 } attrs { promote: false } } }
I20240502 21:25:53.930065 27863 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 25affb8859f94f308ff433d9cb288de0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 21:25:53.931027 27875 catalog_manager.cc:5484] T 25affb8859f94f308ff433d9cb288de0 P 538c626600cd419bb5b9eeae5ded7610 reported cstate change: config changed from index 5 to 7, VOTER e0444722e0404faf9e70415e169e464f (127.24.226.129) evicted. New cstate: current_term: 2 leader_uuid: "538c626600cd419bb5b9eeae5ded7610" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "538c626600cd419bb5b9eeae5ded7610" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 38491 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:53.933835 28000 tablet_service.cc:1514] Processing DeleteTablet for tablet 25affb8859f94f308ff433d9cb288de0 with delete_type TABLET_DATA_TOMBSTONED (TS e0444722e0404faf9e70415e169e464f not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:44274
I20240502 21:25:53.934201 30491 tablet_replica.cc:330] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f: stopping tablet replica
I20240502 21:25:53.934365 30491 raft_consensus.cc:2229] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:25:53.934573 30491 raft_consensus.cc:2258] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:53.934530 29746 consensus_queue.cc:237] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P 18d49d446e634bb5a630d06b99730d21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "18d49d446e634bb5a630d06b99730d21" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 33083 } attrs { promote: false } }
I20240502 21:25:53.935621 30491 ts_tablet_manager.cc:1875] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:25:53.935388 30336 raft_consensus.cc:2932] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P 18d49d446e634bb5a630d06b99730d21 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e0444722e0404faf9e70415e169e464f (127.24.226.129) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "18d49d446e634bb5a630d06b99730d21" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 33083 } attrs { promote: false } } }
I20240502 21:25:53.938966 27865 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 96f3f6d4adfc423e8c2f2ef24ba0cf1f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 21:25:53.940189 27875 catalog_manager.cc:5484] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P 18d49d446e634bb5a630d06b99730d21 reported cstate change: config changed from index 5 to 7, VOTER e0444722e0404faf9e70415e169e464f (127.24.226.129) evicted. New cstate: current_term: 2 leader_uuid: "18d49d446e634bb5a630d06b99730d21" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "18d49d446e634bb5a630d06b99730d21" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 33083 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:53.941102 30491 ts_tablet_manager.cc:1888] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 21:25:53.941227 30491 log.cc:1196] T 25affb8859f94f308ff433d9cb288de0 P e0444722e0404faf9e70415e169e464f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/25affb8859f94f308ff433d9cb288de0
I20240502 21:25:53.941902 27863 catalog_manager.cc:4830] TS e0444722e0404faf9e70415e169e464f (127.24.226.129:39509): tablet 25affb8859f94f308ff433d9cb288de0 (table rebalance_test_table_3 [id=22e3b2f360454a519b85e699fa7dd2d4]) successfully deleted
I20240502 21:25:53.943150 28000 tablet_service.cc:1514] Processing DeleteTablet for tablet 96f3f6d4adfc423e8c2f2ef24ba0cf1f with delete_type TABLET_DATA_TOMBSTONED (TS e0444722e0404faf9e70415e169e464f not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:44274
I20240502 21:25:53.943506 30491 tablet_replica.cc:330] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f: stopping tablet replica
I20240502 21:25:53.943670 30491 raft_consensus.cc:2229] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:25:53.943894 30491 raft_consensus.cc:2258] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:53.944002 29615 consensus_queue.cc:237] T 6a7eaa487ef4426292192a0f4768567c P 6174b3a926654ec291571611f865116d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6174b3a926654ec291571611f865116d" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 37323 } attrs { promote: false } }
I20240502 21:25:53.944686 30491 ts_tablet_manager.cc:1875] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:25:53.944975 30428 raft_consensus.cc:2932] T 6a7eaa487ef4426292192a0f4768567c P 6174b3a926654ec291571611f865116d [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e0444722e0404faf9e70415e169e464f (127.24.226.129) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6174b3a926654ec291571611f865116d" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 37323 } attrs { promote: false } } }
I20240502 21:25:53.948853 27865 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6a7eaa487ef4426292192a0f4768567c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 21:25:53.950150 27875 catalog_manager.cc:5484] T 6a7eaa487ef4426292192a0f4768567c P 6174b3a926654ec291571611f865116d reported cstate change: config changed from index 5 to 7, VOTER e0444722e0404faf9e70415e169e464f (127.24.226.129) evicted. New cstate: current_term: 2 leader_uuid: "6174b3a926654ec291571611f865116d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6174b3a926654ec291571611f865116d" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 37323 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:53.950358 30491 ts_tablet_manager.cc:1888] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 21:25:53.950522 30491 log.cc:1196] T 96f3f6d4adfc423e8c2f2ef24ba0cf1f P e0444722e0404faf9e70415e169e464f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/96f3f6d4adfc423e8c2f2ef24ba0cf1f
I20240502 21:25:53.951241 27863 catalog_manager.cc:4830] TS e0444722e0404faf9e70415e169e464f (127.24.226.129:39509): tablet 96f3f6d4adfc423e8c2f2ef24ba0cf1f (table rebalance_test_table_3 [id=22e3b2f360454a519b85e699fa7dd2d4]) successfully deleted
I20240502 21:25:53.952989 28000 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a7eaa487ef4426292192a0f4768567c with delete_type TABLET_DATA_TOMBSTONED (TS e0444722e0404faf9e70415e169e464f not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:44274
I20240502 21:25:53.953354 30491 tablet_replica.cc:330] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f: stopping tablet replica
I20240502 21:25:53.953510 30491 raft_consensus.cc:2229] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:25:53.953656 30491 raft_consensus.cc:2258] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:53.954385 30491 ts_tablet_manager.cc:1875] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:25:53.956116 29615 consensus_queue.cc:237] T 6d31b0bb7992486f834cb67cd90eb74b P 6174b3a926654ec291571611f865116d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6174b3a926654ec291571611f865116d" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 37323 } attrs { promote: false } }
I20240502 21:25:53.956827 30428 raft_consensus.cc:2932] T 6d31b0bb7992486f834cb67cd90eb74b P 6174b3a926654ec291571611f865116d [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 37e9c08ca35f419b874cf64812057324 (127.24.226.130) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6174b3a926654ec291571611f865116d" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 37323 } attrs { promote: false } } }
I20240502 21:25:53.959486 30491 ts_tablet_manager.cc:1888] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 21:25:53.959642 30491 log.cc:1196] T 6a7eaa487ef4426292192a0f4768567c P e0444722e0404faf9e70415e169e464f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/6a7eaa487ef4426292192a0f4768567c
I20240502 21:25:53.960144 27865 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6d31b0bb7992486f834cb67cd90eb74b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 21:25:53.960357 27863 catalog_manager.cc:4830] TS e0444722e0404faf9e70415e169e464f (127.24.226.129:39509): tablet 6a7eaa487ef4426292192a0f4768567c (table rebalance_test_table_2 [id=6fc5508a149a46b6b19e75ddf5817ae1]) successfully deleted
I20240502 21:25:53.961249 27875 catalog_manager.cc:5484] T 6d31b0bb7992486f834cb67cd90eb74b P 6174b3a926654ec291571611f865116d reported cstate change: config changed from index 5 to 7, VOTER 37e9c08ca35f419b874cf64812057324 (127.24.226.130) evicted. New cstate: current_term: 2 leader_uuid: "6174b3a926654ec291571611f865116d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6174b3a926654ec291571611f865116d" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 37323 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:53.963768 28130 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d31b0bb7992486f834cb67cd90eb74b with delete_type TABLET_DATA_TOMBSTONED (TS 37e9c08ca35f419b874cf64812057324 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49646
I20240502 21:25:53.964069 30489 tablet_replica.cc:330] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324: stopping tablet replica
I20240502 21:25:53.964224 30489 raft_consensus.cc:2229] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:25:53.964392 30489 raft_consensus.cc:2258] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:53.965189 30489 ts_tablet_manager.cc:1875] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:25:53.969192 30489 ts_tablet_manager.cc:1888] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 21:25:53.969354 30489 log.cc:1196] T 6d31b0bb7992486f834cb67cd90eb74b P 37e9c08ca35f419b874cf64812057324: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/6d31b0bb7992486f834cb67cd90eb74b
I20240502 21:25:53.969971 27865 catalog_manager.cc:4830] TS 37e9c08ca35f419b874cf64812057324 (127.24.226.130:35721): tablet 6d31b0bb7992486f834cb67cd90eb74b (table rebalance_test_table_4 [id=3bafa5250c0e45519eb4f187c574fce7]) successfully deleted
I20240502 21:25:54.037626 29225 consensus_queue.cc:237] T d3b7a019c07149d9a291267a76909669 P e5b2432ac32b4c8281bfe645caf7aea2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 46635 } attrs { promote: false } }
I20240502 21:25:54.038575 30037 raft_consensus.cc:2932] T d3b7a019c07149d9a291267a76909669 P e5b2432ac32b4c8281bfe645caf7aea2 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e0444722e0404faf9e70415e169e464f (127.24.226.129) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 46635 } attrs { promote: false } } }
I20240502 21:25:54.041472 27862 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d3b7a019c07149d9a291267a76909669 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 21:25:54.042624 27875 catalog_manager.cc:5484] T d3b7a019c07149d9a291267a76909669 P e5b2432ac32b4c8281bfe645caf7aea2 reported cstate change: config changed from index 5 to 7, VOTER e0444722e0404faf9e70415e169e464f (127.24.226.129) evicted. New cstate: current_term: 2 leader_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e5b2432ac32b4c8281bfe645caf7aea2" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 46635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:54.045923 28000 tablet_service.cc:1514] Processing DeleteTablet for tablet d3b7a019c07149d9a291267a76909669 with delete_type TABLET_DATA_TOMBSTONED (TS e0444722e0404faf9e70415e169e464f not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:44274
I20240502 21:25:54.046267 30491 tablet_replica.cc:330] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f: stopping tablet replica
I20240502 21:25:54.046456 30491 raft_consensus.cc:2229] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:25:54.046604 30491 raft_consensus.cc:2258] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:25:54.047397 30491 ts_tablet_manager.cc:1875] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:25:54.051836 30491 ts_tablet_manager.cc:1888] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 21:25:54.051923 30491 log.cc:1196] T d3b7a019c07149d9a291267a76909669 P e0444722e0404faf9e70415e169e464f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/d3b7a019c07149d9a291267a76909669
I20240502 21:25:54.052559 27863 catalog_manager.cc:4830] TS e0444722e0404faf9e70415e169e464f (127.24.226.129:39509): tablet d3b7a019c07149d9a291267a76909669 (table rebalance_test_table_1 [id=f3b24b58d96b4bceb7b44fc6fe502ad0]) successfully deleted
I20240502 21:25:54.235137 29466 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:54.237928 29726 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:54.239022 28130 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:25:54.246343 29336 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:54.249249 29205 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:54.249749 28000 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:25:54.254674 29596 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:54.426900 29726 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:54.427805 29466 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:54.432556 28130 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:25:54.439217 28000 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:25:54.454331 29336 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:54.457142 29596 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:54.464675 29205 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:55.092942 29466 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:55.093760 28130 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:25:55.095412 29726 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:55.138996 29596 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:55.164608 29336 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:55.190692 28000 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:25:55.192983 29205 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:55.341223 28130 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:25:55.343657 29726 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:55.347692 29466 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:55.359874 29596 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:55.363580 29336 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:55.384541 29205 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:55.385826 28000 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:25:55.640156 29466 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:55.648046 28130 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:25:55.649765 29726 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:55.650861 29596 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:55.672807 29336 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:55.691327 28000 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:25:55.709689 29205 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
5dc89f2545c14a249317777eb9782c83 | 127.24.226.190:32867 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.24.226.148:40945 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
18d49d446e634bb5a630d06b99730d21 | 127.24.226.135:33083 | HEALTHY | <none> | 2 | 0
37e9c08ca35f419b874cf64812057324 | 127.24.226.130:35721 | HEALTHY | <none> | 3 | 0
538c626600cd419bb5b9eeae5ded7610 | 127.24.226.133:38491 | HEALTHY | <none> | 2 | 0
6174b3a926654ec291571611f865116d | 127.24.226.134:37323 | HEALTHY | <none> | 2 | 0
7b66fa9f55994d0caf5dc7d5a8189b2f | 127.24.226.132:38475 | HEALTHY | <none> | 2 | 0
e0444722e0404faf9e70415e169e464f | 127.24.226.129:39509 | HEALTHY | <none> | 3 | 0
e5b2432ac32b4c8281bfe645caf7aea2 | 127.24.226.131:46635 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.24.226.129 | experimental | 127.24.226.129:39509
local_ip_for_outbound_sockets | 127.24.226.130 | experimental | 127.24.226.130:35721
local_ip_for_outbound_sockets | 127.24.226.131 | experimental | 127.24.226.131:46635
local_ip_for_outbound_sockets | 127.24.226.132 | experimental | 127.24.226.132:38475
local_ip_for_outbound_sockets | 127.24.226.133 | experimental | 127.24.226.133:38491
local_ip_for_outbound_sockets | 127.24.226.134 | experimental | 127.24.226.134:37323
local_ip_for_outbound_sockets | 127.24.226.135 | experimental | 127.24.226.135:33083
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.24.226.129:39509
server_dump_info_path | /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.24.226.130:35721
server_dump_info_path | /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.24.226.131:46635
server_dump_info_path | /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.24.226.132:38475
server_dump_info_path | /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.24.226.133:38491
server_dump_info_path | /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.24.226.134:37323
server_dump_info_path | /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.24.226.135:33083
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.24.226.148:40945 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 21:25:55.758984 25482 log_verifier.cc:126] Checking tablet 0357275420cb49d799d26033f9804bef
I20240502 21:25:55.773452 25482 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 0357275420cb49d799d26033f9804bef
I20240502 21:25:55.773589 25482 log_verifier.cc:126] Checking tablet 25affb8859f94f308ff433d9cb288de0
I20240502 21:25:55.785228 25482 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 25affb8859f94f308ff433d9cb288de0
I20240502 21:25:55.785311 25482 log_verifier.cc:126] Checking tablet 4aeae3dbc5f7474dba8ed8841feab6c3
I20240502 21:25:55.796350 25482 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 4aeae3dbc5f7474dba8ed8841feab6c3
I20240502 21:25:55.796442 25482 log_verifier.cc:126] Checking tablet 53fea94f8310494b9354f09fc92ca06a
I20240502 21:25:55.807730 25482 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 53fea94f8310494b9354f09fc92ca06a
I20240502 21:25:55.807821 25482 log_verifier.cc:126] Checking tablet 6a7eaa487ef4426292192a0f4768567c
I20240502 21:25:55.821996 25482 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 6a7eaa487ef4426292192a0f4768567c
I20240502 21:25:55.822127 25482 log_verifier.cc:126] Checking tablet 6cccd964394b451c9d602223e94acd82
I20240502 21:25:55.836179 25482 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6cccd964394b451c9d602223e94acd82
I20240502 21:25:55.836284 25482 log_verifier.cc:126] Checking tablet 6d31b0bb7992486f834cb67cd90eb74b
I20240502 21:25:55.850014 25482 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 6d31b0bb7992486f834cb67cd90eb74b
I20240502 21:25:55.850114 25482 log_verifier.cc:126] Checking tablet 805c2e7913f34cbd883482be9583fa3c
I20240502 21:25:55.861488 25482 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 805c2e7913f34cbd883482be9583fa3c
I20240502 21:25:55.861567 25482 log_verifier.cc:126] Checking tablet 92f8d60108924e42866b1f35c3673443
I20240502 21:25:55.872349 25482 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 92f8d60108924e42866b1f35c3673443
I20240502 21:25:55.872447 25482 log_verifier.cc:126] Checking tablet 96f3f6d4adfc423e8c2f2ef24ba0cf1f
I20240502 21:25:55.888377 25482 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 96f3f6d4adfc423e8c2f2ef24ba0cf1f
I20240502 21:25:55.888505 25482 log_verifier.cc:126] Checking tablet c3aa757477ad450ab8d4c3029b16c670
I20240502 21:25:55.902954 25482 log_verifier.cc:177] Verified matching terms for 7 ops in tablet c3aa757477ad450ab8d4c3029b16c670
I20240502 21:25:55.903038 25482 log_verifier.cc:126] Checking tablet c607e33cdf814d00b3d005493d941a0a
I20240502 21:25:55.920365 25482 log_verifier.cc:177] Verified matching terms for 7 ops in tablet c607e33cdf814d00b3d005493d941a0a
I20240502 21:25:55.920508 25482 log_verifier.cc:126] Checking tablet d3b7a019c07149d9a291267a76909669
I20240502 21:25:55.940455 25482 log_verifier.cc:177] Verified matching terms for 7 ops in tablet d3b7a019c07149d9a291267a76909669
I20240502 21:25:55.940591 25482 log_verifier.cc:126] Checking tablet e18547d3903940c1ab603b8d4129b6d3
I20240502 21:25:55.956655 25482 log_verifier.cc:177] Verified matching terms for 5 ops in tablet e18547d3903940c1ab603b8d4129b6d3
I20240502 21:25:55.956743 25482 log_verifier.cc:126] Checking tablet e5fb7a69b82c42678d35301469f999ea
I20240502 21:25:55.971076 25482 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e5fb7a69b82c42678d35301469f999ea
I20240502 21:25:55.971164 25482 log_verifier.cc:126] Checking tablet fe34567313ce48c1a0caeb2b306bce57
I20240502 21:25:55.986689 25482 log_verifier.cc:177] Verified matching terms for 3 ops in tablet fe34567313ce48c1a0caeb2b306bce57
I20240502 21:25:55.989256 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.136:0
--local_ip_for_outbound_sockets=127.24.226.136
--webserver_interface=127.24.226.136
--webserver_port=0
--tserver_master_addrs=127.24.226.190:32867
--builtin_ntp_servers=127.24.226.148:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 21:25:56.335183 30789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:56.335433 30789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:56.335592 30789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:56.343109 30789 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:56.343267 30789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:56.343426 30789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.136
I20240502 21:25:56.352346 30789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.24.226.136
--webserver_port=0
--tserver_master_addrs=127.24.226.190:32867
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.136
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:56.353613 30789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:56.355319 30789 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:56.367228 30795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:56.367336 30794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:56.367270 30797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:56.367949 30789 server_base.cc:1034] running on GCE node
I20240502 21:25:56.368516 30789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:56.369344 30789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:56.370543 30789 hybrid_clock.cc:648] HybridClock initialized: now 1714685156370484 us; error 40 us; skew 500 ppm
I20240502 21:25:56.370781 30789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:56.372828 30789 webserver.cc:460] Webserver started at http://127.24.226.136:37149/ using document root <none> and password file <none>
I20240502 21:25:56.373437 30789 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:56.373528 30789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:56.373780 30789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:56.376055 30789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "c451af9fb361484388a7210900b12a1e"
format_stamp: "Formatted at 2024-05-02 21:25:56 on dist-test-slave-41k1"
I20240502 21:25:56.376749 30789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "c451af9fb361484388a7210900b12a1e"
format_stamp: "Formatted at 2024-05-02 21:25:56 on dist-test-slave-41k1"
I20240502 21:25:56.381604 30789 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 21:25:56.384828 30803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:56.386013 30789 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20240502 21:25:56.386152 30789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "c451af9fb361484388a7210900b12a1e"
format_stamp: "Formatted at 2024-05-02 21:25:56 on dist-test-slave-41k1"
I20240502 21:25:56.386332 30789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:56.407122 30789 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:56.408033 30789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:56.408378 30789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:56.409359 30789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:56.410948 30789 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:56.411055 30789 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:56.411168 30789 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:56.411247 30789 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:56.453088 30789 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.136:42787
I20240502 21:25:56.453181 30915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.136:42787 every 8 connection(s)
I20240502 21:25:56.454468 30789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 21:25:56.456938 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 30789
I20240502 21:25:56.457173 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 21:25:56.477613 30916 heartbeater.cc:347] Connected to a master server at 127.24.226.190:32867
I20240502 21:25:56.477867 30916 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:56.478680 30916 heartbeater.cc:511] Master 127.24.226.190:32867 requested a full tablet report, sending...
I20240502 21:25:56.480125 27876 ts_manager.cc:194] Registered new tserver with Master: c451af9fb361484388a7210900b12a1e (127.24.226.136:42787)
I20240502 21:25:56.481329 27876 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.136:42751
I20240502 21:25:56.954166 29726 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:56.956960 28130 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:25:56.974872 30851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:57.011023 29596 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:57.028360 29336 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:57.040997 29466 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:57.067802 28000 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:25:57.072295 29205 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:57.260387 28130 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:25:57.261723 30851 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:57.267604 28000 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:25:57.270085 29336 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:57.279258 29205 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:57.279803 29726 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:57.289093 29466 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:57.304720 29596 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:57.378144 28151 consensus_queue.cc:237] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "c451af9fb361484388a7210900b12a1e" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 42787 } attrs { promote: true } }
I20240502 21:25:57.380028 31010 raft_consensus.cc:2932] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c451af9fb361484388a7210900b12a1e (127.24.226.136) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "c451af9fb361484388a7210900b12a1e" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 42787 } attrs { promote: true } } }
I20240502 21:25:57.385187 27876 catalog_manager.cc:5484] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324 reported cstate change: config changed from index -1 to 4, NON_VOTER c451af9fb361484388a7210900b12a1e (127.24.226.136) added. New cstate: current_term: 1 leader_uuid: "37e9c08ca35f419b874cf64812057324" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c451af9fb361484388a7210900b12a1e" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 42787 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 21:25:57.387750 28084 consensus_peers.cc:485] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324 -> Peer c451af9fb361484388a7210900b12a1e (127.24.226.136:42787): Couldn't send request to peer c451af9fb361484388a7210900b12a1e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe34567313ce48c1a0caeb2b306bce57. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:57.395416 28019 consensus_queue.cc:237] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "c451af9fb361484388a7210900b12a1e" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 42787 } attrs { promote: true } }
I20240502 21:25:57.397138 31023 raft_consensus.cc:2932] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c451af9fb361484388a7210900b12a1e (127.24.226.136) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "c451af9fb361484388a7210900b12a1e" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 42787 } attrs { promote: true } } }
I20240502 21:25:57.401961 27876 catalog_manager.cc:5484] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f reported cstate change: config changed from index -1 to 4, NON_VOTER c451af9fb361484388a7210900b12a1e (127.24.226.136) added. New cstate: current_term: 1 leader_uuid: "e0444722e0404faf9e70415e169e464f" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c451af9fb361484388a7210900b12a1e" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 42787 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 21:25:57.404727 27953 consensus_peers.cc:485] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f -> Peer c451af9fb361484388a7210900b12a1e (127.24.226.136:42787): Couldn't send request to peer c451af9fb361484388a7210900b12a1e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0357275420cb49d799d26033f9804bef. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:57.483490 30916 heartbeater.cc:503] Master 127.24.226.190:32867 was elected leader, sending a full tablet report...
I20240502 21:25:57.866741 31056 ts_tablet_manager.cc:902] T fe34567313ce48c1a0caeb2b306bce57 P c451af9fb361484388a7210900b12a1e: Initiating tablet copy from peer 37e9c08ca35f419b874cf64812057324 (127.24.226.130:35721)
I20240502 21:25:57.868412 31056 tablet_copy_client.cc:308] T fe34567313ce48c1a0caeb2b306bce57 P c451af9fb361484388a7210900b12a1e: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.130:35721
I20240502 21:25:57.874312 28171 tablet_copy_service.cc:140] P 37e9c08ca35f419b874cf64812057324: Received BeginTabletCopySession request for tablet fe34567313ce48c1a0caeb2b306bce57 from peer c451af9fb361484388a7210900b12a1e ({username='slave'} at 127.24.226.136:38047)
I20240502 21:25:57.874454 28171 tablet_copy_service.cc:161] P 37e9c08ca35f419b874cf64812057324: Beginning new tablet copy session on tablet fe34567313ce48c1a0caeb2b306bce57 from peer c451af9fb361484388a7210900b12a1e at {username='slave'} at 127.24.226.136:38047: session id = c451af9fb361484388a7210900b12a1e-fe34567313ce48c1a0caeb2b306bce57
I20240502 21:25:57.876211 28171 tablet_copy_source_session.cc:205] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:25:57.880188 31056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe34567313ce48c1a0caeb2b306bce57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:57.888695 31056 tablet_copy_client.cc:783] T fe34567313ce48c1a0caeb2b306bce57 P c451af9fb361484388a7210900b12a1e: tablet copy: Starting download of 0 data blocks...
I20240502 21:25:57.888965 31056 tablet_copy_client.cc:647] T fe34567313ce48c1a0caeb2b306bce57 P c451af9fb361484388a7210900b12a1e: tablet copy: Starting download of 1 WAL segments...
I20240502 21:25:57.890815 31056 tablet_copy_client.cc:523] T fe34567313ce48c1a0caeb2b306bce57 P c451af9fb361484388a7210900b12a1e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:25:57.893386 31056 tablet_bootstrap.cc:492] T fe34567313ce48c1a0caeb2b306bce57 P c451af9fb361484388a7210900b12a1e: Bootstrap starting.
I20240502 21:25:57.902732 31056 log.cc:824] T fe34567313ce48c1a0caeb2b306bce57 P c451af9fb361484388a7210900b12a1e: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:57.920311 31064 ts_tablet_manager.cc:902] T 0357275420cb49d799d26033f9804bef P c451af9fb361484388a7210900b12a1e: Initiating tablet copy from peer e0444722e0404faf9e70415e169e464f (127.24.226.129:39509)
I20240502 21:25:57.921914 31064 tablet_copy_client.cc:308] T 0357275420cb49d799d26033f9804bef P c451af9fb361484388a7210900b12a1e: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.129:39509
I20240502 21:25:57.927325 28040 tablet_copy_service.cc:140] P e0444722e0404faf9e70415e169e464f: Received BeginTabletCopySession request for tablet 0357275420cb49d799d26033f9804bef from peer c451af9fb361484388a7210900b12a1e ({username='slave'} at 127.24.226.136:60333)
I20240502 21:25:57.927476 28040 tablet_copy_service.cc:161] P e0444722e0404faf9e70415e169e464f: Beginning new tablet copy session on tablet 0357275420cb49d799d26033f9804bef from peer c451af9fb361484388a7210900b12a1e at {username='slave'} at 127.24.226.136:60333: session id = c451af9fb361484388a7210900b12a1e-0357275420cb49d799d26033f9804bef
I20240502 21:25:57.930192 28040 tablet_copy_source_session.cc:205] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:25:57.931306 31064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0357275420cb49d799d26033f9804bef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:57.935487 31064 tablet_copy_client.cc:783] T 0357275420cb49d799d26033f9804bef P c451af9fb361484388a7210900b12a1e: tablet copy: Starting download of 0 data blocks...
I20240502 21:25:57.935712 31064 tablet_copy_client.cc:647] T 0357275420cb49d799d26033f9804bef P c451af9fb361484388a7210900b12a1e: tablet copy: Starting download of 1 WAL segments...
I20240502 21:25:57.936446 31056 tablet_bootstrap.cc:492] T fe34567313ce48c1a0caeb2b306bce57 P c451af9fb361484388a7210900b12a1e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:25:57.936815 31056 tablet_bootstrap.cc:492] T fe34567313ce48c1a0caeb2b306bce57 P c451af9fb361484388a7210900b12a1e: Bootstrap complete.
I20240502 21:25:57.937173 31056 ts_tablet_manager.cc:1367] T fe34567313ce48c1a0caeb2b306bce57 P c451af9fb361484388a7210900b12a1e: Time spent bootstrapping tablet: real 0.044s user 0.037s sys 0.007s
I20240502 21:25:57.937209 31064 tablet_copy_client.cc:523] T 0357275420cb49d799d26033f9804bef P c451af9fb361484388a7210900b12a1e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:25:57.939616 31064 tablet_bootstrap.cc:492] T 0357275420cb49d799d26033f9804bef P c451af9fb361484388a7210900b12a1e: Bootstrap starting.
I20240502 21:25:57.942302 31056 raft_consensus.cc:348] T fe34567313ce48c1a0caeb2b306bce57 P c451af9fb361484388a7210900b12a1e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "c451af9fb361484388a7210900b12a1e" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 42787 } attrs { promote: true } }
I20240502 21:25:57.942726 31056 raft_consensus.cc:729] T fe34567313ce48c1a0caeb2b306bce57 P c451af9fb361484388a7210900b12a1e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c451af9fb361484388a7210900b12a1e, State: Initialized, Role: LEARNER
I20240502 21:25:57.943274 31056 consensus_queue.cc:260] T fe34567313ce48c1a0caeb2b306bce57 P c451af9fb361484388a7210900b12a1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "c451af9fb361484388a7210900b12a1e" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 42787 } attrs { promote: true } }
I20240502 21:25:57.944775 31056 ts_tablet_manager.cc:1398] T fe34567313ce48c1a0caeb2b306bce57 P c451af9fb361484388a7210900b12a1e: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240502 21:25:57.945622 28171 tablet_copy_service.cc:342] P 37e9c08ca35f419b874cf64812057324: Request end of tablet copy session c451af9fb361484388a7210900b12a1e-fe34567313ce48c1a0caeb2b306bce57 received from {username='slave'} at 127.24.226.136:38047
I20240502 21:25:57.945734 28171 tablet_copy_service.cc:433] P 37e9c08ca35f419b874cf64812057324: ending tablet copy session c451af9fb361484388a7210900b12a1e-fe34567313ce48c1a0caeb2b306bce57 on tablet fe34567313ce48c1a0caeb2b306bce57 with peer c451af9fb361484388a7210900b12a1e
I20240502 21:25:57.971748 31064 tablet_bootstrap.cc:492] T 0357275420cb49d799d26033f9804bef P c451af9fb361484388a7210900b12a1e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:25:57.972026 31064 tablet_bootstrap.cc:492] T 0357275420cb49d799d26033f9804bef P c451af9fb361484388a7210900b12a1e: Bootstrap complete.
I20240502 21:25:57.972255 31064 ts_tablet_manager.cc:1367] T 0357275420cb49d799d26033f9804bef P c451af9fb361484388a7210900b12a1e: Time spent bootstrapping tablet: real 0.033s user 0.023s sys 0.008s
I20240502 21:25:57.972990 31064 raft_consensus.cc:348] T 0357275420cb49d799d26033f9804bef P c451af9fb361484388a7210900b12a1e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "c451af9fb361484388a7210900b12a1e" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 42787 } attrs { promote: true } }
I20240502 21:25:57.973155 31064 raft_consensus.cc:729] T 0357275420cb49d799d26033f9804bef P c451af9fb361484388a7210900b12a1e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c451af9fb361484388a7210900b12a1e, State: Initialized, Role: LEARNER
I20240502 21:25:57.973323 31064 consensus_queue.cc:260] T 0357275420cb49d799d26033f9804bef P c451af9fb361484388a7210900b12a1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "c451af9fb361484388a7210900b12a1e" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 42787 } attrs { promote: true } }
I20240502 21:25:57.973743 31064 ts_tablet_manager.cc:1398] T 0357275420cb49d799d26033f9804bef P c451af9fb361484388a7210900b12a1e: Time spent starting tablet: real 0.001s user 0.000s sys 0.004s
I20240502 21:25:57.974381 28040 tablet_copy_service.cc:342] P e0444722e0404faf9e70415e169e464f: Request end of tablet copy session c451af9fb361484388a7210900b12a1e-0357275420cb49d799d26033f9804bef received from {username='slave'} at 127.24.226.136:60333
I20240502 21:25:57.974498 28040 tablet_copy_service.cc:433] P e0444722e0404faf9e70415e169e464f: ending tablet copy session c451af9fb361484388a7210900b12a1e-0357275420cb49d799d26033f9804bef on tablet 0357275420cb49d799d26033f9804bef with peer c451af9fb361484388a7210900b12a1e
I20240502 21:25:58.358597 30871 raft_consensus.cc:1203] T fe34567313ce48c1a0caeb2b306bce57 P c451af9fb361484388a7210900b12a1e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:25:58.453631 30871 raft_consensus.cc:1203] T 0357275420cb49d799d26033f9804bef P c451af9fb361484388a7210900b12a1e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:25:58.894665 31117 raft_consensus.cc:1050] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324: attempting to promote NON_VOTER c451af9fb361484388a7210900b12a1e to VOTER
I20240502 21:25:58.895224 31117 consensus_queue.cc:237] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "c451af9fb361484388a7210900b12a1e" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 42787 } attrs { promote: false } }
I20240502 21:25:58.896800 30871 raft_consensus.cc:1261] T fe34567313ce48c1a0caeb2b306bce57 P c451af9fb361484388a7210900b12a1e [term 1 LEARNER]: Refusing update from remote peer 37e9c08ca35f419b874cf64812057324: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:25:58.897300 31117 consensus_queue.cc:1035] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c451af9fb361484388a7210900b12a1e" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 42787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:25:58.900198 31116 raft_consensus.cc:2932] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c451af9fb361484388a7210900b12a1e (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "c451af9fb361484388a7210900b12a1e" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 42787 } attrs { promote: false } } }
I20240502 21:25:58.900897 30871 raft_consensus.cc:2932] T fe34567313ce48c1a0caeb2b306bce57 P c451af9fb361484388a7210900b12a1e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c451af9fb361484388a7210900b12a1e (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "c451af9fb361484388a7210900b12a1e" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 42787 } attrs { promote: false } } }
I20240502 21:25:58.904280 27876 catalog_manager.cc:5484] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324 reported cstate change: config changed from index 4 to 5, c451af9fb361484388a7210900b12a1e (127.24.226.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "37e9c08ca35f419b874cf64812057324" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c451af9fb361484388a7210900b12a1e" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 42787 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:58.946453 28151 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fe34567313ce48c1a0caeb2b306bce57"
dest_uuid: "37e9c08ca35f419b874cf64812057324"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48004
I20240502 21:25:58.946619 28151 raft_consensus.cc:595] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:25:59.071900 31140 raft_consensus.cc:1050] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f: attempting to promote NON_VOTER c451af9fb361484388a7210900b12a1e to VOTER
I20240502 21:25:59.072515 31140 consensus_queue.cc:237] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "c451af9fb361484388a7210900b12a1e" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 42787 } attrs { promote: false } }
I20240502 21:25:59.073989 30871 raft_consensus.cc:1261] T 0357275420cb49d799d26033f9804bef P c451af9fb361484388a7210900b12a1e [term 1 LEARNER]: Refusing update from remote peer e0444722e0404faf9e70415e169e464f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:25:59.074452 31140 consensus_queue.cc:1035] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c451af9fb361484388a7210900b12a1e" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 42787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:25:59.076790 31141 raft_consensus.cc:2932] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c451af9fb361484388a7210900b12a1e (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "c451af9fb361484388a7210900b12a1e" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 42787 } attrs { promote: false } } }
I20240502 21:25:59.077391 30871 raft_consensus.cc:2932] T 0357275420cb49d799d26033f9804bef P c451af9fb361484388a7210900b12a1e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c451af9fb361484388a7210900b12a1e (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "c451af9fb361484388a7210900b12a1e" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 42787 } attrs { promote: false } } }
I20240502 21:25:59.080736 27870 catalog_manager.cc:5484] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f reported cstate change: config changed from index 4 to 5, c451af9fb361484388a7210900b12a1e (127.24.226.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e0444722e0404faf9e70415e169e464f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c451af9fb361484388a7210900b12a1e" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 42787 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:59.163892 28019 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0357275420cb49d799d26033f9804bef"
dest_uuid: "e0444722e0404faf9e70415e169e464f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59800
I20240502 21:25:59.164081 28019 raft_consensus.cc:595] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Received request to transfer leadership
I20240502 21:25:59.179591 28151 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fe34567313ce48c1a0caeb2b306bce57"
dest_uuid: "37e9c08ca35f419b874cf64812057324"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48020
I20240502 21:25:59.179836 28151 raft_consensus.cc:595] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:25:59.367969 31117 raft_consensus.cc:979] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324: : Instructing follower c451af9fb361484388a7210900b12a1e to start an election
I20240502 21:25:59.368176 31117 raft_consensus.cc:1067] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Signalling peer c451af9fb361484388a7210900b12a1e to start an election
I20240502 21:25:59.368928 30871 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fe34567313ce48c1a0caeb2b306bce57"
dest_uuid: "c451af9fb361484388a7210900b12a1e"
from {username='slave'} at 127.24.226.130:53817
I20240502 21:25:59.369194 30871 raft_consensus.cc:482] T fe34567313ce48c1a0caeb2b306bce57 P c451af9fb361484388a7210900b12a1e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:25:59.369383 30871 raft_consensus.cc:3037] T fe34567313ce48c1a0caeb2b306bce57 P c451af9fb361484388a7210900b12a1e [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:59.373558 30871 raft_consensus.cc:504] T fe34567313ce48c1a0caeb2b306bce57 P c451af9fb361484388a7210900b12a1e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "c451af9fb361484388a7210900b12a1e" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 42787 } attrs { promote: false } }
I20240502 21:25:59.374751 30871 leader_election.cc:288] T fe34567313ce48c1a0caeb2b306bce57 P c451af9fb361484388a7210900b12a1e [CANDIDATE]: Term 2 election: Requested vote from peers 37e9c08ca35f419b874cf64812057324 (127.24.226.130:35721)
I20240502 21:25:59.375221 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe34567313ce48c1a0caeb2b306bce57" candidate_uuid: "c451af9fb361484388a7210900b12a1e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "37e9c08ca35f419b874cf64812057324"
I20240502 21:25:59.375453 28151 raft_consensus.cc:3032] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:25:59.375540 28151 raft_consensus.cc:729] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 37e9c08ca35f419b874cf64812057324, State: Running, Role: LEADER
I20240502 21:25:59.375794 28151 consensus_queue.cc:260] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "c451af9fb361484388a7210900b12a1e" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 42787 } attrs { promote: false } }
I20240502 21:25:59.376040 28151 raft_consensus.cc:3037] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:59.377718 28151 raft_consensus.cc:2454] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c451af9fb361484388a7210900b12a1e in term 2.
I20240502 21:25:59.378201 30807 leader_election.cc:302] T fe34567313ce48c1a0caeb2b306bce57 P c451af9fb361484388a7210900b12a1e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 37e9c08ca35f419b874cf64812057324, c451af9fb361484388a7210900b12a1e; no voters:
I20240502 21:25:59.378538 31069 raft_consensus.cc:2781] T fe34567313ce48c1a0caeb2b306bce57 P c451af9fb361484388a7210900b12a1e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:25:59.378887 31069 raft_consensus.cc:686] T fe34567313ce48c1a0caeb2b306bce57 P c451af9fb361484388a7210900b12a1e [term 2 LEADER]: Becoming Leader. State: Replica: c451af9fb361484388a7210900b12a1e, State: Running, Role: LEADER
I20240502 21:25:59.379293 31069 consensus_queue.cc:237] T fe34567313ce48c1a0caeb2b306bce57 P c451af9fb361484388a7210900b12a1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } } peers { permanent_uuid: "c451af9fb361484388a7210900b12a1e" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 42787 } attrs { promote: false } }
I20240502 21:25:59.382524 27870 catalog_manager.cc:5484] T fe34567313ce48c1a0caeb2b306bce57 P c451af9fb361484388a7210900b12a1e reported cstate change: term changed from 1 to 2, leader changed from 37e9c08ca35f419b874cf64812057324 (127.24.226.130) to c451af9fb361484388a7210900b12a1e (127.24.226.136). New cstate: current_term: 2 leader_uuid: "c451af9fb361484388a7210900b12a1e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c451af9fb361484388a7210900b12a1e" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 42787 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 21:25:59.394639 27862 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet fe34567313ce48c1a0caeb2b306bce57 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader c451af9fb361484388a7210900b12a1e (127.24.226.136:42787) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240502 21:25:59.399161 28019 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0357275420cb49d799d26033f9804bef"
dest_uuid: "e0444722e0404faf9e70415e169e464f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59818
I20240502 21:25:59.399381 28019 raft_consensus.cc:595] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Received request to transfer leadership
I20240502 21:25:59.583737 31182 raft_consensus.cc:979] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f: : Instructing follower c451af9fb361484388a7210900b12a1e to start an election
I20240502 21:25:59.583891 31182 raft_consensus.cc:1067] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Signalling peer c451af9fb361484388a7210900b12a1e to start an election
I20240502 21:25:59.584540 30871 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0357275420cb49d799d26033f9804bef"
dest_uuid: "c451af9fb361484388a7210900b12a1e"
from {username='slave'} at 127.24.226.129:39221
I20240502 21:25:59.584730 30871 raft_consensus.cc:482] T 0357275420cb49d799d26033f9804bef P c451af9fb361484388a7210900b12a1e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:25:59.584818 30871 raft_consensus.cc:3037] T 0357275420cb49d799d26033f9804bef P c451af9fb361484388a7210900b12a1e [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:59.586571 30871 raft_consensus.cc:504] T 0357275420cb49d799d26033f9804bef P c451af9fb361484388a7210900b12a1e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "c451af9fb361484388a7210900b12a1e" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 42787 } attrs { promote: false } }
I20240502 21:25:59.587046 30871 leader_election.cc:288] T 0357275420cb49d799d26033f9804bef P c451af9fb361484388a7210900b12a1e [CANDIDATE]: Term 2 election: Requested vote from peers e0444722e0404faf9e70415e169e464f (127.24.226.129:39509)
I20240502 21:25:59.587513 28019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0357275420cb49d799d26033f9804bef" candidate_uuid: "c451af9fb361484388a7210900b12a1e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e0444722e0404faf9e70415e169e464f"
I20240502 21:25:59.587695 28019 raft_consensus.cc:3032] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:25:59.587800 28019 raft_consensus.cc:729] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e0444722e0404faf9e70415e169e464f, State: Running, Role: LEADER
I20240502 21:25:59.588007 28019 consensus_queue.cc:260] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "c451af9fb361484388a7210900b12a1e" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 42787 } attrs { promote: false } }
I20240502 21:25:59.588274 28019 raft_consensus.cc:3037] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:25:59.590380 28019 raft_consensus.cc:2454] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c451af9fb361484388a7210900b12a1e in term 2.
I20240502 21:25:59.590852 30805 leader_election.cc:302] T 0357275420cb49d799d26033f9804bef P c451af9fb361484388a7210900b12a1e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c451af9fb361484388a7210900b12a1e, e0444722e0404faf9e70415e169e464f; no voters:
I20240502 21:25:59.591152 31069 raft_consensus.cc:2781] T 0357275420cb49d799d26033f9804bef P c451af9fb361484388a7210900b12a1e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:25:59.591321 31069 raft_consensus.cc:686] T 0357275420cb49d799d26033f9804bef P c451af9fb361484388a7210900b12a1e [term 2 LEADER]: Becoming Leader. State: Replica: c451af9fb361484388a7210900b12a1e, State: Running, Role: LEADER
I20240502 21:25:59.591559 31069 consensus_queue.cc:237] T 0357275420cb49d799d26033f9804bef P c451af9fb361484388a7210900b12a1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } } peers { permanent_uuid: "c451af9fb361484388a7210900b12a1e" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 42787 } attrs { promote: false } }
I20240502 21:25:59.593600 27870 catalog_manager.cc:5484] T 0357275420cb49d799d26033f9804bef P c451af9fb361484388a7210900b12a1e reported cstate change: term changed from 1 to 2, leader changed from e0444722e0404faf9e70415e169e464f (127.24.226.129) to c451af9fb361484388a7210900b12a1e (127.24.226.136). New cstate: current_term: 2 leader_uuid: "c451af9fb361484388a7210900b12a1e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c451af9fb361484388a7210900b12a1e" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 42787 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:25:59.974066 28151 raft_consensus.cc:1261] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324 [term 2 FOLLOWER]: Refusing update from remote peer c451af9fb361484388a7210900b12a1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:25:59.974709 31069 consensus_queue.cc:1035] T fe34567313ce48c1a0caeb2b306bce57 P c451af9fb361484388a7210900b12a1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "37e9c08ca35f419b874cf64812057324" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 35721 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:00.050122 30871 consensus_queue.cc:237] T fe34567313ce48c1a0caeb2b306bce57 P c451af9fb361484388a7210900b12a1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c451af9fb361484388a7210900b12a1e" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 42787 } attrs { promote: false } }
I20240502 21:26:00.051182 31208 raft_consensus.cc:2932] T fe34567313ce48c1a0caeb2b306bce57 P c451af9fb361484388a7210900b12a1e [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 37e9c08ca35f419b874cf64812057324 (127.24.226.130) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c451af9fb361484388a7210900b12a1e" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 42787 } attrs { promote: false } } }
I20240502 21:26:00.054435 27862 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fe34567313ce48c1a0caeb2b306bce57 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 21:26:00.055840 27870 catalog_manager.cc:5484] T fe34567313ce48c1a0caeb2b306bce57 P c451af9fb361484388a7210900b12a1e reported cstate change: config changed from index 5 to 7, VOTER 37e9c08ca35f419b874cf64812057324 (127.24.226.130) evicted. New cstate: current_term: 2 leader_uuid: "c451af9fb361484388a7210900b12a1e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c451af9fb361484388a7210900b12a1e" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 42787 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:00.059621 28130 tablet_service.cc:1514] Processing DeleteTablet for tablet fe34567313ce48c1a0caeb2b306bce57 with delete_type TABLET_DATA_TOMBSTONED (TS 37e9c08ca35f419b874cf64812057324 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49646
I20240502 21:26:00.060225 31216 tablet_replica.cc:330] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324: stopping tablet replica
I20240502 21:26:00.060415 31216 raft_consensus.cc:2229] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:00.060669 31216 raft_consensus.cc:2258] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:00.062441 31216 ts_tablet_manager.cc:1875] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:00.067471 31216 ts_tablet_manager.cc:1888] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 21:26:00.067600 31216 log.cc:1196] T fe34567313ce48c1a0caeb2b306bce57 P 37e9c08ca35f419b874cf64812057324: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/fe34567313ce48c1a0caeb2b306bce57
I20240502 21:26:00.068300 27865 catalog_manager.cc:4830] TS 37e9c08ca35f419b874cf64812057324 (127.24.226.130:35721): tablet fe34567313ce48c1a0caeb2b306bce57 (table rebalance_test_table_3 [id=22e3b2f360454a519b85e699fa7dd2d4]) successfully deleted
I20240502 21:26:00.127964 28019 raft_consensus.cc:1261] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f [term 2 FOLLOWER]: Refusing update from remote peer c451af9fb361484388a7210900b12a1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:26:00.128531 31208 consensus_queue.cc:1035] T 0357275420cb49d799d26033f9804bef P c451af9fb361484388a7210900b12a1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0444722e0404faf9e70415e169e464f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 39509 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:00.190260 30871 consensus_queue.cc:237] T 0357275420cb49d799d26033f9804bef P c451af9fb361484388a7210900b12a1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c451af9fb361484388a7210900b12a1e" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 42787 } attrs { promote: false } }
I20240502 21:26:00.191073 31069 raft_consensus.cc:2932] T 0357275420cb49d799d26033f9804bef P c451af9fb361484388a7210900b12a1e [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e0444722e0404faf9e70415e169e464f (127.24.226.129) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c451af9fb361484388a7210900b12a1e" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 42787 } attrs { promote: false } } }
I20240502 21:26:00.193495 27862 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0357275420cb49d799d26033f9804bef with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 21:26:00.194531 27870 catalog_manager.cc:5484] T 0357275420cb49d799d26033f9804bef P c451af9fb361484388a7210900b12a1e reported cstate change: config changed from index 5 to 7, VOTER e0444722e0404faf9e70415e169e464f (127.24.226.129) evicted. New cstate: current_term: 2 leader_uuid: "c451af9fb361484388a7210900b12a1e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c451af9fb361484388a7210900b12a1e" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 42787 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:00.197304 28000 tablet_service.cc:1514] Processing DeleteTablet for tablet 0357275420cb49d799d26033f9804bef with delete_type TABLET_DATA_TOMBSTONED (TS e0444722e0404faf9e70415e169e464f not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:44274
I20240502 21:26:00.197815 31228 tablet_replica.cc:330] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f: stopping tablet replica
I20240502 21:26:00.197979 31228 raft_consensus.cc:2229] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:00.198164 31228 raft_consensus.cc:2258] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:00.199645 31228 ts_tablet_manager.cc:1875] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:00.203683 31228 ts_tablet_manager.cc:1888] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 21:26:00.203794 31228 log.cc:1196] T 0357275420cb49d799d26033f9804bef P e0444722e0404faf9e70415e169e464f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/0357275420cb49d799d26033f9804bef
I20240502 21:26:00.204423 27863 catalog_manager.cc:4830] TS e0444722e0404faf9e70415e169e464f (127.24.226.129:39509): tablet 0357275420cb49d799d26033f9804bef (table rebalance_test_table_1 [id=f3b24b58d96b4bceb7b44fc6fe502ad0]) successfully deleted
I20240502 21:26:00.379825 29466 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:00.384481 29726 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:00.387615 28130 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:00.402426 29336 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:00.413314 30851 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:00.445473 29596 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:00.457947 29205 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:00.475190 28000 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:00.635644 29466 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:00.642645 29726 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:00.644639 28130 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:00.645036 29596 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:00.650547 30851 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:00.650940 29336 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:00.656630 29205 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:00.672662 28000 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:00.960672 29726 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:00.969549 29466 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:00.972689 29596 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:00.988772 30851 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:00.997081 29205 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:00.997515 28130 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:01.001399 28000 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:01.015715 29336 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
5dc89f2545c14a249317777eb9782c83 | 127.24.226.190:32867 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.24.226.148:40945 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
18d49d446e634bb5a630d06b99730d21 | 127.24.226.135:33083 | HEALTHY | <none> | 2 | 0
37e9c08ca35f419b874cf64812057324 | 127.24.226.130:35721 | HEALTHY | <none> | 2 | 0
538c626600cd419bb5b9eeae5ded7610 | 127.24.226.133:38491 | HEALTHY | <none> | 2 | 0
6174b3a926654ec291571611f865116d | 127.24.226.134:37323 | HEALTHY | <none> | 2 | 0
7b66fa9f55994d0caf5dc7d5a8189b2f | 127.24.226.132:38475 | HEALTHY | <none> | 2 | 0
c451af9fb361484388a7210900b12a1e | 127.24.226.136:42787 | HEALTHY | <none> | 2 | 0
e0444722e0404faf9e70415e169e464f | 127.24.226.129:39509 | HEALTHY | <none> | 2 | 0
e5b2432ac32b4c8281bfe645caf7aea2 | 127.24.226.131:46635 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.24.226.129 | experimental | 127.24.226.129:39509
local_ip_for_outbound_sockets | 127.24.226.130 | experimental | 127.24.226.130:35721
local_ip_for_outbound_sockets | 127.24.226.131 | experimental | 127.24.226.131:46635
local_ip_for_outbound_sockets | 127.24.226.132 | experimental | 127.24.226.132:38475
local_ip_for_outbound_sockets | 127.24.226.133 | experimental | 127.24.226.133:38491
local_ip_for_outbound_sockets | 127.24.226.134 | experimental | 127.24.226.134:37323
local_ip_for_outbound_sockets | 127.24.226.135 | experimental | 127.24.226.135:33083
local_ip_for_outbound_sockets | 127.24.226.136 | experimental | 127.24.226.136:42787
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.24.226.129:39509
server_dump_info_path | /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.24.226.130:35721
server_dump_info_path | /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.24.226.131:46635
server_dump_info_path | /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.24.226.132:38475
server_dump_info_path | /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.24.226.133:38491
server_dump_info_path | /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.24.226.134:37323
server_dump_info_path | /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.24.226.135:33083
server_dump_info_path | /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.24.226.136:42787
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.24.226.148:40945 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 21:26:01.092134 25482 log_verifier.cc:126] Checking tablet 0357275420cb49d799d26033f9804bef
I20240502 21:26:01.105146 25482 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 0357275420cb49d799d26033f9804bef
I20240502 21:26:01.105266 25482 log_verifier.cc:126] Checking tablet 25affb8859f94f308ff433d9cb288de0
I20240502 21:26:01.119637 25482 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 25affb8859f94f308ff433d9cb288de0
I20240502 21:26:01.119722 25482 log_verifier.cc:126] Checking tablet 4aeae3dbc5f7474dba8ed8841feab6c3
I20240502 21:26:01.133373 25482 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 4aeae3dbc5f7474dba8ed8841feab6c3
I20240502 21:26:01.133455 25482 log_verifier.cc:126] Checking tablet 53fea94f8310494b9354f09fc92ca06a
I20240502 21:26:01.142551 25482 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 53fea94f8310494b9354f09fc92ca06a
I20240502 21:26:01.142634 25482 log_verifier.cc:126] Checking tablet 6a7eaa487ef4426292192a0f4768567c
I20240502 21:26:01.151896 25482 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 6a7eaa487ef4426292192a0f4768567c
I20240502 21:26:01.151954 25482 log_verifier.cc:126] Checking tablet 6cccd964394b451c9d602223e94acd82
I20240502 21:26:01.160215 25482 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6cccd964394b451c9d602223e94acd82
I20240502 21:26:01.160297 25482 log_verifier.cc:126] Checking tablet 6d31b0bb7992486f834cb67cd90eb74b
I20240502 21:26:01.170269 25482 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 6d31b0bb7992486f834cb67cd90eb74b
I20240502 21:26:01.170339 25482 log_verifier.cc:126] Checking tablet 805c2e7913f34cbd883482be9583fa3c
I20240502 21:26:01.179190 25482 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 805c2e7913f34cbd883482be9583fa3c
I20240502 21:26:01.179248 25482 log_verifier.cc:126] Checking tablet 92f8d60108924e42866b1f35c3673443
I20240502 21:26:01.187911 25482 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 92f8d60108924e42866b1f35c3673443
I20240502 21:26:01.187974 25482 log_verifier.cc:126] Checking tablet 96f3f6d4adfc423e8c2f2ef24ba0cf1f
I20240502 21:26:01.197959 25482 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 96f3f6d4adfc423e8c2f2ef24ba0cf1f
I20240502 21:26:01.198030 25482 log_verifier.cc:126] Checking tablet c3aa757477ad450ab8d4c3029b16c670
I20240502 21:26:01.207003 25482 log_verifier.cc:177] Verified matching terms for 7 ops in tablet c3aa757477ad450ab8d4c3029b16c670
I20240502 21:26:01.207118 25482 log_verifier.cc:126] Checking tablet c607e33cdf814d00b3d005493d941a0a
I20240502 21:26:01.216295 25482 log_verifier.cc:177] Verified matching terms for 7 ops in tablet c607e33cdf814d00b3d005493d941a0a
I20240502 21:26:01.216374 25482 log_verifier.cc:126] Checking tablet d3b7a019c07149d9a291267a76909669
I20240502 21:26:01.225643 25482 log_verifier.cc:177] Verified matching terms for 7 ops in tablet d3b7a019c07149d9a291267a76909669
I20240502 21:26:01.225713 25482 log_verifier.cc:126] Checking tablet e18547d3903940c1ab603b8d4129b6d3
I20240502 21:26:01.231531 25482 log_verifier.cc:177] Verified matching terms for 5 ops in tablet e18547d3903940c1ab603b8d4129b6d3
I20240502 21:26:01.231608 25482 log_verifier.cc:126] Checking tablet e5fb7a69b82c42678d35301469f999ea
I20240502 21:26:01.239940 25482 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e5fb7a69b82c42678d35301469f999ea
I20240502 21:26:01.240010 25482 log_verifier.cc:126] Checking tablet fe34567313ce48c1a0caeb2b306bce57
I20240502 21:26:01.251323 25482 log_verifier.cc:177] Verified matching terms for 7 ops in tablet fe34567313ce48c1a0caeb2b306bce57
I20240502 21:26:01.252178 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 27938
I20240502 21:26:01.253624 28060 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:01.386472 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 27938
I20240502 21:26:01.411165 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 28069
I20240502 21:26:01.412150 28190 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:01.549633 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 28069
I20240502 21:26:01.574345 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 29143
I20240502 21:26:01.579910 29265 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:01.686774 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 29143
I20240502 21:26:01.706393 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 29274
I20240502 21:26:01.711982 29395 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:01.816913 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 29274
W20240502 21:26:01.838047 27863 connection.cc:535] server connection from 127.24.226.132:41179 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:26:01.838256 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 29404
I20240502 21:26:01.843835 29526 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:01.955631 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 29404
W20240502 21:26:01.975114 27862 connection.cc:535] server connection from 127.24.226.133:33873 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:26:01.975404 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 29534
I20240502 21:26:01.980803 29655 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:02.086730 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 29534
W20240502 21:26:02.106215 27865 connection.cc:535] server connection from 127.24.226.134:33947 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:26:02.106480 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 29664
I20240502 21:26:02.112139 29786 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:02.219180 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 29664
I20240502 21:26:02.239113 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 30789
I20240502 21:26:02.244331 30910 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:02.335791 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 30789
I20240502 21:26:02.354101 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 27847
I20240502 21:26:02.355123 27908 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:02.498801 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 27847
2024-05-02T21:26:02Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/1 (24880 ms)
[----------] 1 test from RebalanceParamTest (24880 ms total)
[----------] 1 test from DDLDuringRebalancingTest
[ RUN ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/1
I20240502 21:26:02.564299 25482 test_util.cc:268] Using random seed: 1074014903
I20240502 21:26:02.565932 25482 ts_itest-base.cc:112] Starting cluster with:
I20240502 21:26:02.565984 25482 ts_itest-base.cc:113] --------------
I20240502 21:26:02.566021 25482 ts_itest-base.cc:114] 8 tablet servers
I20240502 21:26:02.566051 25482 ts_itest-base.cc:115] 3 replicas per TS
I20240502 21:26:02.566109 25482 ts_itest-base.cc:116] --------------
2024-05-02T21:26:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T21:26:02Z Disabled control of system clock
I20240502 21:26:02.595492 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.226.190:43531
--webserver_interface=127.24.226.190
--webserver_port=0
--builtin_ntp_servers=127.24.226.148:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.226.190:43531
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 21:26:02.889932 31364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:02.890201 31364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:02.890326 31364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:02.896719 31364 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:02.896824 31364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 21:26:02.896879 31364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:02.896960 31364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 21:26:02.897018 31364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 21:26:02.905169 31364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.24.226.190:43531
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.226.190:43531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.24.226.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:02.906136 31364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:02.907552 31364 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:02.917012 31370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:02.916980 31372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:02.917038 31369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:02.917469 31364 server_base.cc:1034] running on GCE node
I20240502 21:26:02.918064 31364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:02.919148 31364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:02.920336 31364 hybrid_clock.cc:648] HybridClock initialized: now 1714685162920295 us; error 30 us; skew 500 ppm
I20240502 21:26:02.920683 31364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:02.922613 31364 webserver.cc:460] Webserver started at http://127.24.226.190:36795/ using document root <none> and password file <none>
I20240502 21:26:02.923193 31364 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:02.923295 31364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:02.923549 31364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:02.925705 31364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "5afb5e5a489743fe980538330e79474e"
format_stamp: "Formatted at 2024-05-02 21:26:02 on dist-test-slave-41k1"
I20240502 21:26:02.926354 31364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "5afb5e5a489743fe980538330e79474e"
format_stamp: "Formatted at 2024-05-02 21:26:02 on dist-test-slave-41k1"
I20240502 21:26:02.930609 31364 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 21:26:02.933351 31378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:02.934331 31364 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:26:02.934486 31364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/wal
uuid: "5afb5e5a489743fe980538330e79474e"
format_stamp: "Formatted at 2024-05-02 21:26:02 on dist-test-slave-41k1"
I20240502 21:26:02.934651 31364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:02.960112 31364 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:02.960824 31364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:02.961086 31364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:02.977694 31364 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.190:43531
I20240502 21:26:02.977720 31429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.190:43531 every 8 connection(s)
I20240502 21:26:02.978783 31364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 21:26:02.979375 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 31364
I20240502 21:26:02.979555 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 21:26:02.982720 31430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:02.993963 31430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Bootstrap starting.
I20240502 21:26:02.996387 31430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:02.997413 31430 log.cc:824] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:02.999567 31430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: No bootstrap required, opened a new log
I20240502 21:26:03.005085 31430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5afb5e5a489743fe980538330e79474e" member_type: VOTER last_known_addr { host: "127.24.226.190" port: 43531 } }
I20240502 21:26:03.005333 31430 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:03.005410 31430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5afb5e5a489743fe980538330e79474e, State: Initialized, Role: FOLLOWER
I20240502 21:26:03.005857 31430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5afb5e5a489743fe980538330e79474e" member_type: VOTER last_known_addr { host: "127.24.226.190" port: 43531 } }
I20240502 21:26:03.006021 31430 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:26:03.006122 31430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:26:03.006245 31430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:03.008159 31430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5afb5e5a489743fe980538330e79474e" member_type: VOTER last_known_addr { host: "127.24.226.190" port: 43531 } }
I20240502 21:26:03.008669 31430 leader_election.cc:302] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5afb5e5a489743fe980538330e79474e; no voters:
I20240502 21:26:03.009248 31430 leader_election.cc:288] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:26:03.009362 31435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:03.009980 31435 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e [term 1 LEADER]: Becoming Leader. State: Replica: 5afb5e5a489743fe980538330e79474e, State: Running, Role: LEADER
I20240502 21:26:03.010504 31435 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5afb5e5a489743fe980538330e79474e" member_type: VOTER last_known_addr { host: "127.24.226.190" port: 43531 } }
I20240502 21:26:03.010973 31430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:26:03.012676 31437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5afb5e5a489743fe980538330e79474e. Latest consensus state: current_term: 1 leader_uuid: "5afb5e5a489743fe980538330e79474e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5afb5e5a489743fe980538330e79474e" member_type: VOTER last_known_addr { host: "127.24.226.190" port: 43531 } } }
I20240502 21:26:03.012902 31437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e [sys.catalog]: This master's current role is: LEADER
I20240502 21:26:03.013293 31436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5afb5e5a489743fe980538330e79474e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5afb5e5a489743fe980538330e79474e" member_type: VOTER last_known_addr { host: "127.24.226.190" port: 43531 } } }
I20240502 21:26:03.013510 31436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e [sys.catalog]: This master's current role is: LEADER
I20240502 21:26:03.017658 31441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:26:03.022861 31441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:26:03.029716 31441 catalog_manager.cc:1347] Generated new cluster ID: cfef563835224e0ba8518589fd7e41df
I20240502 21:26:03.029814 31441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:26:03.063618 31441 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:26:03.065430 31441 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:26:03.083413 31441 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Generated new TSK 0
I20240502 21:26:03.084326 31441 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:26:03.102031 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.129:0
--local_ip_for_outbound_sockets=127.24.226.129
--webserver_interface=127.24.226.129
--webserver_port=0
--tserver_master_addrs=127.24.226.190:43531
--builtin_ntp_servers=127.24.226.148:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 21:26:03.406472 31454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:03.406733 31454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:03.406889 31454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:03.413786 31454 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:03.413911 31454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:03.414058 31454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.129
I20240502 21:26:03.422219 31454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.24.226.129
--webserver_port=0
--tserver_master_addrs=127.24.226.190:43531
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.129
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:03.423336 31454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:03.424930 31454 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:03.435541 31462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:03.435612 31460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:03.435603 31459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:03.436630 31454 server_base.cc:1034] running on GCE node
I20240502 21:26:03.437124 31454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:03.438011 31454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:03.439217 31454 hybrid_clock.cc:648] HybridClock initialized: now 1714685163439166 us; error 40 us; skew 500 ppm
I20240502 21:26:03.439492 31454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:03.441535 31454 webserver.cc:460] Webserver started at http://127.24.226.129:34185/ using document root <none> and password file <none>
I20240502 21:26:03.442184 31454 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:03.442287 31454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:03.442576 31454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:03.444993 31454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "a44f41b0b6644fb6aa29476aed8c964f"
format_stamp: "Formatted at 2024-05-02 21:26:03 on dist-test-slave-41k1"
I20240502 21:26:03.445643 31454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "a44f41b0b6644fb6aa29476aed8c964f"
format_stamp: "Formatted at 2024-05-02 21:26:03 on dist-test-slave-41k1"
I20240502 21:26:03.450143 31454 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 21:26:03.453109 31468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:03.454170 31454 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:26:03.454313 31454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "a44f41b0b6644fb6aa29476aed8c964f"
format_stamp: "Formatted at 2024-05-02 21:26:03 on dist-test-slave-41k1"
I20240502 21:26:03.454483 31454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:03.472383 31454 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:03.473047 31454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:03.473277 31454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:03.474016 31454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:03.475231 31454 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:03.475308 31454 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:03.475379 31454 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:03.475433 31454 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:03.505681 31454 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.129:41039
I20240502 21:26:03.505726 31580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.129:41039 every 8 connection(s)
I20240502 21:26:03.506758 31454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 21:26:03.513916 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 31454
I20240502 21:26:03.514107 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 21:26:03.517150 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.130:0
--local_ip_for_outbound_sockets=127.24.226.130
--webserver_interface=127.24.226.130
--webserver_port=0
--tserver_master_addrs=127.24.226.190:43531
--builtin_ntp_servers=127.24.226.148:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:26:03.518635 31581 heartbeater.cc:347] Connected to a master server at 127.24.226.190:43531
I20240502 21:26:03.518911 31581 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:03.519780 31581 heartbeater.cc:511] Master 127.24.226.190:43531 requested a full tablet report, sending...
I20240502 21:26:03.521816 31395 ts_manager.cc:194] Registered new tserver with Master: a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:03.523291 31395 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.129:42389
W20240502 21:26:03.808631 31585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:03.808892 31585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:03.809043 31585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:03.817458 31585 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:03.817598 31585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:03.817781 31585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.130
I20240502 21:26:03.826144 31585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.24.226.130
--webserver_port=0
--tserver_master_addrs=127.24.226.190:43531
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.130
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:03.827232 31585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:03.828759 31585 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:03.838872 31591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:03.839015 31590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:03.839310 31593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:03.839704 31585 server_base.cc:1034] running on GCE node
I20240502 21:26:03.840232 31585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:03.841091 31585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:03.842293 31585 hybrid_clock.cc:648] HybridClock initialized: now 1714685163842264 us; error 30 us; skew 500 ppm
I20240502 21:26:03.842553 31585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:03.844353 31585 webserver.cc:460] Webserver started at http://127.24.226.130:40633/ using document root <none> and password file <none>
I20240502 21:26:03.845009 31585 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:03.845113 31585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:03.845386 31585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:03.847509 31585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "05f4fac41c4f4959b9b316cb112d0ada"
format_stamp: "Formatted at 2024-05-02 21:26:03 on dist-test-slave-41k1"
I20240502 21:26:03.848151 31585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "05f4fac41c4f4959b9b316cb112d0ada"
format_stamp: "Formatted at 2024-05-02 21:26:03 on dist-test-slave-41k1"
I20240502 21:26:03.852627 31585 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 21:26:03.855252 31599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:03.856276 31585 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:26:03.856405 31585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "05f4fac41c4f4959b9b316cb112d0ada"
format_stamp: "Formatted at 2024-05-02 21:26:03 on dist-test-slave-41k1"
I20240502 21:26:03.856608 31585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:03.872627 31585 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:03.873247 31585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:03.873466 31585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:03.874256 31585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:03.875448 31585 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:03.875533 31585 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:03.875633 31585 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:03.875696 31585 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:03.905342 31585 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.130:37645
I20240502 21:26:03.905401 31711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.130:37645 every 8 connection(s)
I20240502 21:26:03.906487 31585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 21:26:03.916539 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 31585
I20240502 21:26:03.916718 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 21:26:03.917670 31712 heartbeater.cc:347] Connected to a master server at 127.24.226.190:43531
I20240502 21:26:03.917941 31712 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:03.918722 31712 heartbeater.cc:511] Master 127.24.226.190:43531 requested a full tablet report, sending...
I20240502 21:26:03.919723 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.131:0
--local_ip_for_outbound_sockets=127.24.226.131
--webserver_interface=127.24.226.131
--webserver_port=0
--tserver_master_addrs=127.24.226.190:43531
--builtin_ntp_servers=127.24.226.148:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:26:03.919975 31395 ts_manager.cc:194] Registered new tserver with Master: 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:03.920797 31395 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.130:49213
W20240502 21:26:04.226044 31715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:04.226287 31715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:04.226369 31715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:04.232898 31715 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:04.232988 31715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:04.233081 31715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.131
I20240502 21:26:04.241165 31715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.24.226.131
--webserver_port=0
--tserver_master_addrs=127.24.226.190:43531
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.131
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:04.242179 31715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:04.243604 31715 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:04.253996 31720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:04.254040 31723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:04.254060 31721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:04.254673 31715 server_base.cc:1034] running on GCE node
I20240502 21:26:04.255107 31715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:04.255808 31715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:04.256974 31715 hybrid_clock.cc:648] HybridClock initialized: now 1714685164256935 us; error 32 us; skew 500 ppm
I20240502 21:26:04.257215 31715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:04.259152 31715 webserver.cc:460] Webserver started at http://127.24.226.131:41751/ using document root <none> and password file <none>
I20240502 21:26:04.259753 31715 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:04.259857 31715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:04.260167 31715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:04.262459 31715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "08d7a0680d244b55b93d17f9ab57b84c"
format_stamp: "Formatted at 2024-05-02 21:26:04 on dist-test-slave-41k1"
I20240502 21:26:04.263046 31715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "08d7a0680d244b55b93d17f9ab57b84c"
format_stamp: "Formatted at 2024-05-02 21:26:04 on dist-test-slave-41k1"
I20240502 21:26:04.267546 31715 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20240502 21:26:04.270551 31729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:04.271766 31715 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:04.271894 31715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "08d7a0680d244b55b93d17f9ab57b84c"
format_stamp: "Formatted at 2024-05-02 21:26:04 on dist-test-slave-41k1"
I20240502 21:26:04.272059 31715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:04.287757 31715 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:04.288411 31715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:04.288648 31715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:04.289458 31715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:04.290758 31715 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:04.290843 31715 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:04.290943 31715 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:04.291011 31715 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:04.320004 31715 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.131:40807
I20240502 21:26:04.320078 31841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.131:40807 every 8 connection(s)
I20240502 21:26:04.321259 31715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 21:26:04.328729 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 31715
I20240502 21:26:04.328903 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 21:26:04.331995 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.132:0
--local_ip_for_outbound_sockets=127.24.226.132
--webserver_interface=127.24.226.132
--webserver_port=0
--tserver_master_addrs=127.24.226.190:43531
--builtin_ntp_servers=127.24.226.148:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:26:04.332211 31842 heartbeater.cc:347] Connected to a master server at 127.24.226.190:43531
I20240502 21:26:04.332466 31842 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:04.333123 31842 heartbeater.cc:511] Master 127.24.226.190:43531 requested a full tablet report, sending...
I20240502 21:26:04.334267 31395 ts_manager.cc:194] Registered new tserver with Master: 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:04.335000 31395 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.131:50027
I20240502 21:26:04.526371 31581 heartbeater.cc:503] Master 127.24.226.190:43531 was elected leader, sending a full tablet report...
W20240502 21:26:04.630465 31845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:04.630715 31845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:04.630800 31845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:04.637071 31845 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:04.637217 31845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:04.637318 31845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.132
I20240502 21:26:04.645074 31845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.24.226.132
--webserver_port=0
--tserver_master_addrs=127.24.226.190:43531
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.132
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:04.646078 31845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:04.647568 31845 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:04.657346 31851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:04.657341 31850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:04.658200 31853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:04.658807 31845 server_base.cc:1034] running on GCE node
I20240502 21:26:04.659279 31845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:04.660100 31845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:04.661324 31845 hybrid_clock.cc:648] HybridClock initialized: now 1714685164661266 us; error 43 us; skew 500 ppm
I20240502 21:26:04.661592 31845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:04.663470 31845 webserver.cc:460] Webserver started at http://127.24.226.132:41533/ using document root <none> and password file <none>
I20240502 21:26:04.664175 31845 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:04.664284 31845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:04.664592 31845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:04.666759 31845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "2d50f33557944bb480ea32199d9ee4f1"
format_stamp: "Formatted at 2024-05-02 21:26:04 on dist-test-slave-41k1"
I20240502 21:26:04.667407 31845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "2d50f33557944bb480ea32199d9ee4f1"
format_stamp: "Formatted at 2024-05-02 21:26:04 on dist-test-slave-41k1"
I20240502 21:26:04.671922 31845 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20240502 21:26:04.674729 31859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:04.675782 31845 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:04.675910 31845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "2d50f33557944bb480ea32199d9ee4f1"
format_stamp: "Formatted at 2024-05-02 21:26:04 on dist-test-slave-41k1"
I20240502 21:26:04.676062 31845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:04.706564 31845 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:04.707197 31845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:04.707423 31845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:04.708184 31845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:04.709429 31845 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:04.709504 31845 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:04.709604 31845 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:04.709666 31845 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:04.737262 31845 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.132:35933
I20240502 21:26:04.737315 31971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.132:35933 every 8 connection(s)
I20240502 21:26:04.738391 31845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 21:26:04.739951 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 31845
I20240502 21:26:04.740118 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 21:26:04.743393 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.133:0
--local_ip_for_outbound_sockets=127.24.226.133
--webserver_interface=127.24.226.133
--webserver_port=0
--tserver_master_addrs=127.24.226.190:43531
--builtin_ntp_servers=127.24.226.148:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:26:04.751185 31972 heartbeater.cc:347] Connected to a master server at 127.24.226.190:43531
I20240502 21:26:04.751449 31972 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:04.752103 31972 heartbeater.cc:511] Master 127.24.226.190:43531 requested a full tablet report, sending...
I20240502 21:26:04.753315 31395 ts_manager.cc:194] Registered new tserver with Master: 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:04.753996 31395 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.132:55731
I20240502 21:26:04.922888 31712 heartbeater.cc:503] Master 127.24.226.190:43531 was elected leader, sending a full tablet report...
W20240502 21:26:05.035341 31975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:05.035635 31975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:05.035777 31975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:05.043298 31975 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:05.043455 31975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:05.043618 31975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.133
I20240502 21:26:05.052580 31975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.24.226.133
--webserver_port=0
--tserver_master_addrs=127.24.226.190:43531
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.133
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:05.053793 31975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:05.055429 31975 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:05.066660 31981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:05.066617 31983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:05.066888 31980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:05.067589 31975 server_base.cc:1034] running on GCE node
I20240502 21:26:05.068058 31975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:05.068964 31975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:05.070163 31975 hybrid_clock.cc:648] HybridClock initialized: now 1714685165070107 us; error 46 us; skew 500 ppm
I20240502 21:26:05.070456 31975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:05.072580 31975 webserver.cc:460] Webserver started at http://127.24.226.133:44979/ using document root <none> and password file <none>
I20240502 21:26:05.073275 31975 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:05.073377 31975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:05.073655 31975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:05.076010 31975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "1e04157b98dd4fad98858c290b73c3db"
format_stamp: "Formatted at 2024-05-02 21:26:05 on dist-test-slave-41k1"
I20240502 21:26:05.076772 31975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "1e04157b98dd4fad98858c290b73c3db"
format_stamp: "Formatted at 2024-05-02 21:26:05 on dist-test-slave-41k1"
I20240502 21:26:05.081450 31975 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20240502 21:26:05.084489 31989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:05.085582 31975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 21:26:05.085716 31975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "1e04157b98dd4fad98858c290b73c3db"
format_stamp: "Formatted at 2024-05-02 21:26:05 on dist-test-slave-41k1"
I20240502 21:26:05.085911 31975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:05.108714 31975 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:05.109448 31975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:05.109699 31975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:05.110551 31975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:05.111951 31975 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:05.112030 31975 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:05.112140 31975 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:05.112202 31975 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:05.141621 31975 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.133:39863
I20240502 21:26:05.141692 32101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.133:39863 every 8 connection(s)
I20240502 21:26:05.142693 31975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:26:05.142973 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 31975
I20240502 21:26:05.143136 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 21:26:05.146688 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.134:0
--local_ip_for_outbound_sockets=127.24.226.134
--webserver_interface=127.24.226.134
--webserver_port=0
--tserver_master_addrs=127.24.226.190:43531
--builtin_ntp_servers=127.24.226.148:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:26:05.153667 32102 heartbeater.cc:347] Connected to a master server at 127.24.226.190:43531
I20240502 21:26:05.153937 32102 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:05.154680 32102 heartbeater.cc:511] Master 127.24.226.190:43531 requested a full tablet report, sending...
I20240502 21:26:05.155907 31395 ts_manager.cc:194] Registered new tserver with Master: 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:26:05.156605 31395 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.133:39745
I20240502 21:26:05.336997 31842 heartbeater.cc:503] Master 127.24.226.190:43531 was elected leader, sending a full tablet report...
W20240502 21:26:05.435958 32105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:05.436172 32105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:05.436244 32105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:05.442657 32105 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:05.442745 32105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:05.442835 32105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.134
I20240502 21:26:05.450139 32105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.24.226.134
--webserver_port=0
--tserver_master_addrs=127.24.226.190:43531
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.134
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:05.451154 32105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:05.452992 32105 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:05.463006 32111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:05.463038 32110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:05.463060 32113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:05.463810 32105 server_base.cc:1034] running on GCE node
I20240502 21:26:05.464488 32105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:05.465260 32105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:05.466444 32105 hybrid_clock.cc:648] HybridClock initialized: now 1714685165466407 us; error 29 us; skew 500 ppm
I20240502 21:26:05.466687 32105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:05.468505 32105 webserver.cc:460] Webserver started at http://127.24.226.134:37149/ using document root <none> and password file <none>
I20240502 21:26:05.469251 32105 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:05.469363 32105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:05.469655 32105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:05.471869 32105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "9dec645b473f40578e726f3e50e219bf"
format_stamp: "Formatted at 2024-05-02 21:26:05 on dist-test-slave-41k1"
I20240502 21:26:05.472534 32105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "9dec645b473f40578e726f3e50e219bf"
format_stamp: "Formatted at 2024-05-02 21:26:05 on dist-test-slave-41k1"
I20240502 21:26:05.476730 32105 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 21:26:05.479401 32119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:05.480572 32105 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:26:05.480731 32105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "9dec645b473f40578e726f3e50e219bf"
format_stamp: "Formatted at 2024-05-02 21:26:05 on dist-test-slave-41k1"
I20240502 21:26:05.480927 32105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:05.497859 32105 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:05.498468 32105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:05.498682 32105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:05.499431 32105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:05.500711 32105 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:05.500798 32105 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:05.500911 32105 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:05.500978 32105 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:05.529453 32105 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.134:43753
I20240502 21:26:05.529495 32231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.134:43753 every 8 connection(s)
I20240502 21:26:05.530619 32105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 21:26:05.534541 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 32105
I20240502 21:26:05.534689 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 21:26:05.537453 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.135:0
--local_ip_for_outbound_sockets=127.24.226.135
--webserver_interface=127.24.226.135
--webserver_port=0
--tserver_master_addrs=127.24.226.190:43531
--builtin_ntp_servers=127.24.226.148:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:26:05.541682 32232 heartbeater.cc:347] Connected to a master server at 127.24.226.190:43531
I20240502 21:26:05.541909 32232 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:05.542562 32232 heartbeater.cc:511] Master 127.24.226.190:43531 requested a full tablet report, sending...
I20240502 21:26:05.543905 31395 ts_manager.cc:194] Registered new tserver with Master: 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:26:05.545059 31395 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.134:38543
I20240502 21:26:05.755995 31972 heartbeater.cc:503] Master 127.24.226.190:43531 was elected leader, sending a full tablet report...
W20240502 21:26:05.823792 32235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:05.824005 32235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:05.824074 32235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:05.830235 32235 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:05.830335 32235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:05.830446 32235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.135
I20240502 21:26:05.837952 32235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.24.226.135
--webserver_port=0
--tserver_master_addrs=127.24.226.190:43531
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.135
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:05.838934 32235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:05.840480 32235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:05.850848 32240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:05.850878 32243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:05.850848 32241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:05.851445 32235 server_base.cc:1034] running on GCE node
I20240502 21:26:05.851907 32235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:05.852695 32235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:05.853865 32235 hybrid_clock.cc:648] HybridClock initialized: now 1714685165853831 us; error 31 us; skew 500 ppm
I20240502 21:26:05.854120 32235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:05.855854 32235 webserver.cc:460] Webserver started at http://127.24.226.135:34145/ using document root <none> and password file <none>
I20240502 21:26:05.856458 32235 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:05.856564 32235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:05.856817 32235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:05.858901 32235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
format_stamp: "Formatted at 2024-05-02 21:26:05 on dist-test-slave-41k1"
I20240502 21:26:05.859521 32235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
format_stamp: "Formatted at 2024-05-02 21:26:05 on dist-test-slave-41k1"
I20240502 21:26:05.863837 32235 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20240502 21:26:05.866501 32249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:05.867468 32235 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:05.867596 32235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
format_stamp: "Formatted at 2024-05-02 21:26:05 on dist-test-slave-41k1"
I20240502 21:26:05.867748 32235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:05.898445 32235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:05.899123 32235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:05.899360 32235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:05.900205 32235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:05.901624 32235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:05.901711 32235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:05.901813 32235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:05.901876 32235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:05.930251 32235 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.135:38569
I20240502 21:26:05.930325 32361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.135:38569 every 8 connection(s)
I20240502 21:26:05.931408 32235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 21:26:05.936616 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 32235
I20240502 21:26:05.936789 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 21:26:05.939374 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.136:0
--local_ip_for_outbound_sockets=127.24.226.136
--webserver_interface=127.24.226.136
--webserver_port=0
--tserver_master_addrs=127.24.226.190:43531
--builtin_ntp_servers=127.24.226.148:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:26:05.942301 32362 heartbeater.cc:347] Connected to a master server at 127.24.226.190:43531
I20240502 21:26:05.942565 32362 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:05.943362 32362 heartbeater.cc:511] Master 127.24.226.190:43531 requested a full tablet report, sending...
I20240502 21:26:05.944622 31395 ts_manager.cc:194] Registered new tserver with Master: 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:05.945258 31395 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.135:35919
I20240502 21:26:06.158686 32102 heartbeater.cc:503] Master 127.24.226.190:43531 was elected leader, sending a full tablet report...
W20240502 21:26:06.238474 32365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:06.238737 32365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:06.238871 32365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:06.245688 32365 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:06.245806 32365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:06.245937 32365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.136
I20240502 21:26:06.254204 32365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.24.226.136
--webserver_port=0
--tserver_master_addrs=127.24.226.190:43531
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.136
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:06.255192 32365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:06.256726 32365 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:06.269251 32371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:06.269335 32370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:06.269999 32373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:06.270408 32365 server_base.cc:1034] running on GCE node
I20240502 21:26:06.270957 32365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:06.271761 32365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:06.272960 32365 hybrid_clock.cc:648] HybridClock initialized: now 1714685166272926 us; error 25 us; skew 500 ppm
I20240502 21:26:06.273216 32365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:06.275069 32365 webserver.cc:460] Webserver started at http://127.24.226.136:40265/ using document root <none> and password file <none>
I20240502 21:26:06.275678 32365 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:06.275759 32365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:06.275974 32365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:26:06.278152 32365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "d1406e92aec84ac585304e57bd37a40d"
format_stamp: "Formatted at 2024-05-02 21:26:06 on dist-test-slave-41k1"
I20240502 21:26:06.278726 32365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "d1406e92aec84ac585304e57bd37a40d"
format_stamp: "Formatted at 2024-05-02 21:26:06 on dist-test-slave-41k1"
I20240502 21:26:06.283322 32365 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 21:26:06.286051 32379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:06.287123 32365 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:06.287245 32365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "d1406e92aec84ac585304e57bd37a40d"
format_stamp: "Formatted at 2024-05-02 21:26:06 on dist-test-slave-41k1"
I20240502 21:26:06.287405 32365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:06.297595 32365 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:06.298238 32365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:06.298467 32365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:06.299244 32365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:06.300544 32365 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:06.300631 32365 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:06.300743 32365 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:06.300814 32365 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:06.329051 32365 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.136:34907
I20240502 21:26:06.329115 32491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.136:34907 every 8 connection(s)
I20240502 21:26:06.330192 32365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 21:26:06.337921 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 32365
I20240502 21:26:06.338055 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 21:26:06.341799 32492 heartbeater.cc:347] Connected to a master server at 127.24.226.190:43531
I20240502 21:26:06.342025 32492 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:06.342671 32492 heartbeater.cc:511] Master 127.24.226.190:43531 requested a full tablet report, sending...
I20240502 21:26:06.343730 31395 ts_manager.cc:194] Registered new tserver with Master: d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:06.344344 31395 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.136:35247
I20240502 21:26:06.353494 25482 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 21:26:06.367489 25482 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 21:26:06.367590 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 31975
I20240502 21:26:06.374594 32097 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:06.446934 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 31975
I20240502 21:26:06.462975 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 32105
I20240502 21:26:06.470156 32227 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:06.543121 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 32105
I20240502 21:26:06.556484 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 32235
I20240502 21:26:06.563679 32357 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:06.636027 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 32235
I20240502 21:26:06.651580 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 32365
I20240502 21:26:06.659220 32487 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:26:06.730445 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 32365
I20240502 21:26:10.497673 31395 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:10.540545 31776 tablet_service.cc:1467] Processing CreateTablet for tablet 174d4efa03b5450ebe34170e9279a390 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d813d81d1b9645bab949b858e177664a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:10.540522 31777 tablet_service.cc:1467] Processing CreateTablet for tablet 0af9582263b842de9f78784ce62cb716 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d813d81d1b9645bab949b858e177664a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:10.542127 31777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0af9582263b842de9f78784ce62cb716. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:10.542806 31776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 174d4efa03b5450ebe34170e9279a390. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:10.546705 31907 tablet_service.cc:1467] Processing CreateTablet for tablet 0af9582263b842de9f78784ce62cb716 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d813d81d1b9645bab949b858e177664a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:10.548604 31907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0af9582263b842de9f78784ce62cb716. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:10.547672 31906 tablet_service.cc:1467] Processing CreateTablet for tablet 9d89305df0454659b691ad23a0ff3392 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d813d81d1b9645bab949b858e177664a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:10.549381 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d89305df0454659b691ad23a0ff3392. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:10.552812 31515 tablet_service.cc:1467] Processing CreateTablet for tablet 174d4efa03b5450ebe34170e9279a390 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d813d81d1b9645bab949b858e177664a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:10.553385 31514 tablet_service.cc:1467] Processing CreateTablet for tablet 9d89305df0454659b691ad23a0ff3392 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d813d81d1b9645bab949b858e177664a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:10.552878 31516 tablet_service.cc:1467] Processing CreateTablet for tablet 0af9582263b842de9f78784ce62cb716 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d813d81d1b9645bab949b858e177664a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:10.554086 31515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 174d4efa03b5450ebe34170e9279a390. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:10.554930 31514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d89305df0454659b691ad23a0ff3392. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:10.555150 31516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0af9582263b842de9f78784ce62cb716. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:10.565491 32520 tablet_bootstrap.cc:492] T 174d4efa03b5450ebe34170e9279a390 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:10.568282 32521 tablet_bootstrap.cc:492] T 9d89305df0454659b691ad23a0ff3392 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:10.568740 32520 tablet_bootstrap.cc:654] T 174d4efa03b5450ebe34170e9279a390 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:10.568799 31647 tablet_service.cc:1467] Processing CreateTablet for tablet 174d4efa03b5450ebe34170e9279a390 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d813d81d1b9645bab949b858e177664a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:10.569864 31647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 174d4efa03b5450ebe34170e9279a390. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:10.569985 32520 log.cc:824] T 174d4efa03b5450ebe34170e9279a390 P 08d7a0680d244b55b93d17f9ab57b84c: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:10.571748 32521 tablet_bootstrap.cc:654] T 9d89305df0454659b691ad23a0ff3392 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:10.572610 32521 log.cc:824] T 9d89305df0454659b691ad23a0ff3392 P 2d50f33557944bb480ea32199d9ee4f1: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:10.568100 31646 tablet_service.cc:1467] Processing CreateTablet for tablet 9d89305df0454659b691ad23a0ff3392 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d813d81d1b9645bab949b858e177664a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:10.573107 31646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d89305df0454659b691ad23a0ff3392. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:10.574610 32520 tablet_bootstrap.cc:492] T 174d4efa03b5450ebe34170e9279a390 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:10.574859 32520 ts_tablet_manager.cc:1367] T 174d4efa03b5450ebe34170e9279a390 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240502 21:26:10.583348 32520 raft_consensus.cc:348] T 174d4efa03b5450ebe34170e9279a390 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:10.583905 32520 raft_consensus.cc:374] T 174d4efa03b5450ebe34170e9279a390 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:10.584308 32520 raft_consensus.cc:729] T 174d4efa03b5450ebe34170e9279a390 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:10.586102 32520 consensus_queue.cc:260] T 174d4efa03b5450ebe34170e9279a390 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:10.594024 32520 ts_tablet_manager.cc:1398] T 174d4efa03b5450ebe34170e9279a390 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.019s user 0.011s sys 0.008s
I20240502 21:26:10.594590 32520 tablet_bootstrap.cc:492] T 0af9582263b842de9f78784ce62cb716 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:10.595875 32525 tablet_bootstrap.cc:492] T 174d4efa03b5450ebe34170e9279a390 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:10.596364 32520 tablet_bootstrap.cc:654] T 0af9582263b842de9f78784ce62cb716 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:10.603190 32525 tablet_bootstrap.cc:654] T 174d4efa03b5450ebe34170e9279a390 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:10.604704 32525 log.cc:824] T 174d4efa03b5450ebe34170e9279a390 P 05f4fac41c4f4959b9b316cb112d0ada: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:10.606462 32520 tablet_bootstrap.cc:492] T 0af9582263b842de9f78784ce62cb716 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:10.606668 32520 ts_tablet_manager.cc:1367] T 0af9582263b842de9f78784ce62cb716 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.006s
I20240502 21:26:10.607685 32520 raft_consensus.cc:348] T 0af9582263b842de9f78784ce62cb716 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:10.607913 32520 raft_consensus.cc:374] T 0af9582263b842de9f78784ce62cb716 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:10.608045 32520 raft_consensus.cc:729] T 0af9582263b842de9f78784ce62cb716 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:10.607820 32528 tablet_bootstrap.cc:492] T 0af9582263b842de9f78784ce62cb716 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:10.608367 32520 consensus_queue.cc:260] T 0af9582263b842de9f78784ce62cb716 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:10.609014 32520 ts_tablet_manager.cc:1398] T 0af9582263b842de9f78784ce62cb716 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:10.613512 32525 tablet_bootstrap.cc:492] T 174d4efa03b5450ebe34170e9279a390 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:10.613730 32525 ts_tablet_manager.cc:1367] T 174d4efa03b5450ebe34170e9279a390 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.005s
I20240502 21:26:10.617671 32528 tablet_bootstrap.cc:654] T 0af9582263b842de9f78784ce62cb716 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:10.617981 32521 tablet_bootstrap.cc:492] T 9d89305df0454659b691ad23a0ff3392 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:10.618342 32521 ts_tablet_manager.cc:1367] T 9d89305df0454659b691ad23a0ff3392 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.050s user 0.017s sys 0.032s
I20240502 21:26:10.619285 32528 log.cc:824] T 0af9582263b842de9f78784ce62cb716 P a44f41b0b6644fb6aa29476aed8c964f: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:10.620334 32525 raft_consensus.cc:348] T 174d4efa03b5450ebe34170e9279a390 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:10.620725 32525 raft_consensus.cc:374] T 174d4efa03b5450ebe34170e9279a390 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:10.620833 32525 raft_consensus.cc:729] T 174d4efa03b5450ebe34170e9279a390 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:10.621410 32525 consensus_queue.cc:260] T 174d4efa03b5450ebe34170e9279a390 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:10.621834 32528 tablet_bootstrap.cc:492] T 0af9582263b842de9f78784ce62cb716 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:10.622093 32528 ts_tablet_manager.cc:1367] T 0af9582263b842de9f78784ce62cb716 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.002s
I20240502 21:26:10.623703 32525 ts_tablet_manager.cc:1398] T 174d4efa03b5450ebe34170e9279a390 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.010s user 0.005s sys 0.003s
I20240502 21:26:10.624152 32525 tablet_bootstrap.cc:492] T 9d89305df0454659b691ad23a0ff3392 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:10.626132 32525 tablet_bootstrap.cc:654] T 9d89305df0454659b691ad23a0ff3392 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:10.626325 32521 raft_consensus.cc:348] T 9d89305df0454659b691ad23a0ff3392 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:10.626679 32521 raft_consensus.cc:374] T 9d89305df0454659b691ad23a0ff3392 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:10.626770 32521 raft_consensus.cc:729] T 9d89305df0454659b691ad23a0ff3392 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:10.627353 32521 consensus_queue.cc:260] T 9d89305df0454659b691ad23a0ff3392 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:10.628842 32521 ts_tablet_manager.cc:1398] T 9d89305df0454659b691ad23a0ff3392 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20240502 21:26:10.629408 32521 tablet_bootstrap.cc:492] T 0af9582263b842de9f78784ce62cb716 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:10.630971 32521 tablet_bootstrap.cc:654] T 0af9582263b842de9f78784ce62cb716 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:10.630787 32528 raft_consensus.cc:348] T 0af9582263b842de9f78784ce62cb716 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:10.631187 32528 raft_consensus.cc:374] T 0af9582263b842de9f78784ce62cb716 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:10.631307 32528 raft_consensus.cc:729] T 0af9582263b842de9f78784ce62cb716 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:10.631932 32528 consensus_queue.cc:260] T 0af9582263b842de9f78784ce62cb716 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:10.633386 32528 ts_tablet_manager.cc:1398] T 0af9582263b842de9f78784ce62cb716 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.011s user 0.004s sys 0.006s
I20240502 21:26:10.633684 32528 tablet_bootstrap.cc:492] T 9d89305df0454659b691ad23a0ff3392 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:10.633828 32525 tablet_bootstrap.cc:492] T 9d89305df0454659b691ad23a0ff3392 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:10.633960 32525 ts_tablet_manager.cc:1367] T 9d89305df0454659b691ad23a0ff3392 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240502 21:26:10.634778 32528 tablet_bootstrap.cc:654] T 9d89305df0454659b691ad23a0ff3392 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:10.634790 32525 raft_consensus.cc:348] T 9d89305df0454659b691ad23a0ff3392 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:10.634967 32525 raft_consensus.cc:374] T 9d89305df0454659b691ad23a0ff3392 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:10.635094 32525 raft_consensus.cc:729] T 9d89305df0454659b691ad23a0ff3392 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:10.635356 32525 consensus_queue.cc:260] T 9d89305df0454659b691ad23a0ff3392 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:10.636344 32521 tablet_bootstrap.cc:492] T 0af9582263b842de9f78784ce62cb716 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:10.636518 32521 ts_tablet_manager.cc:1367] T 0af9582263b842de9f78784ce62cb716 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 21:26:10.636677 32528 tablet_bootstrap.cc:492] T 9d89305df0454659b691ad23a0ff3392 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:10.636945 32528 ts_tablet_manager.cc:1367] T 9d89305df0454659b691ad23a0ff3392 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s
I20240502 21:26:10.637341 32525 ts_tablet_manager.cc:1398] T 9d89305df0454659b691ad23a0ff3392 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:10.637336 32521 raft_consensus.cc:348] T 0af9582263b842de9f78784ce62cb716 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:10.637503 32521 raft_consensus.cc:374] T 0af9582263b842de9f78784ce62cb716 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:10.637578 32521 raft_consensus.cc:729] T 0af9582263b842de9f78784ce62cb716 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:10.637756 32521 consensus_queue.cc:260] T 0af9582263b842de9f78784ce62cb716 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:10.637754 32528 raft_consensus.cc:348] T 9d89305df0454659b691ad23a0ff3392 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:10.637928 32528 raft_consensus.cc:374] T 9d89305df0454659b691ad23a0ff3392 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:10.638003 32528 raft_consensus.cc:729] T 9d89305df0454659b691ad23a0ff3392 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:10.638199 32521 ts_tablet_manager.cc:1398] T 0af9582263b842de9f78784ce62cb716 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:10.638224 32528 consensus_queue.cc:260] T 9d89305df0454659b691ad23a0ff3392 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:10.638815 32528 ts_tablet_manager.cc:1398] T 9d89305df0454659b691ad23a0ff3392 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:10.639076 32528 tablet_bootstrap.cc:492] T 174d4efa03b5450ebe34170e9279a390 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:10.640372 32528 tablet_bootstrap.cc:654] T 174d4efa03b5450ebe34170e9279a390 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:10.642078 32528 tablet_bootstrap.cc:492] T 174d4efa03b5450ebe34170e9279a390 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:10.642220 32528 ts_tablet_manager.cc:1367] T 174d4efa03b5450ebe34170e9279a390 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:10.642666 32528 raft_consensus.cc:348] T 174d4efa03b5450ebe34170e9279a390 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:10.642807 32528 raft_consensus.cc:374] T 174d4efa03b5450ebe34170e9279a390 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:10.642885 32528 raft_consensus.cc:729] T 174d4efa03b5450ebe34170e9279a390 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:10.643074 32528 consensus_queue.cc:260] T 174d4efa03b5450ebe34170e9279a390 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:10.643438 32528 ts_tablet_manager.cc:1398] T 174d4efa03b5450ebe34170e9279a390 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240502 21:26:10.662166 31713 tablet.cc:2369] T 174d4efa03b5450ebe34170e9279a390 P 05f4fac41c4f4959b9b316cb112d0ada: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:10.718767 32534 raft_consensus.cc:482] T 9d89305df0454659b691ad23a0ff3392 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:10.719013 32534 raft_consensus.cc:504] T 9d89305df0454659b691ad23a0ff3392 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:10.720608 32534 leader_election.cc:288] T 9d89305df0454659b691ad23a0ff3392 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:10.726703 31667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d89305df0454659b691ad23a0ff3392" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:10.727212 31667 raft_consensus.cc:2454] T 9d89305df0454659b691ad23a0ff3392 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:10.727646 31927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d89305df0454659b691ad23a0ff3392" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:10.727777 31472 leader_election.cc:302] T 9d89305df0454659b691ad23a0ff3392 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:10.728085 31927 raft_consensus.cc:2454] T 9d89305df0454659b691ad23a0ff3392 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:10.728160 32534 raft_consensus.cc:2781] T 9d89305df0454659b691ad23a0ff3392 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:10.728304 32534 raft_consensus.cc:482] T 9d89305df0454659b691ad23a0ff3392 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:10.728379 32534 raft_consensus.cc:3037] T 9d89305df0454659b691ad23a0ff3392 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:10.730401 32534 raft_consensus.cc:504] T 9d89305df0454659b691ad23a0ff3392 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:10.730943 32534 leader_election.cc:288] T 9d89305df0454659b691ad23a0ff3392 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Requested vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:10.731290 31667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d89305df0454659b691ad23a0ff3392" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:10.731441 31667 raft_consensus.cc:3037] T 9d89305df0454659b691ad23a0ff3392 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:10.731443 31927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d89305df0454659b691ad23a0ff3392" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:10.731603 31927 raft_consensus.cc:3037] T 9d89305df0454659b691ad23a0ff3392 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:10.733338 31667 raft_consensus.cc:2454] T 9d89305df0454659b691ad23a0ff3392 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:10.733613 31927 raft_consensus.cc:2454] T 9d89305df0454659b691ad23a0ff3392 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:10.733650 31472 leader_election.cc:302] T 9d89305df0454659b691ad23a0ff3392 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:10.733927 32534 raft_consensus.cc:2781] T 9d89305df0454659b691ad23a0ff3392 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:10.733999 32524 raft_consensus.cc:482] T 0af9582263b842de9f78784ce62cb716 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:10.734208 32524 raft_consensus.cc:504] T 0af9582263b842de9f78784ce62cb716 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:10.734386 32534 raft_consensus.cc:686] T 9d89305df0454659b691ad23a0ff3392 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:26:10.734813 32534 consensus_queue.cc:237] T 9d89305df0454659b691ad23a0ff3392 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:10.735808 32524 leader_election.cc:288] T 0af9582263b842de9f78784ce62cb716 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:10.740360 31393 catalog_manager.cc:5484] T 9d89305df0454659b691ad23a0ff3392 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 0 to 1, leader changed from <none> to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } }
W20240502 21:26:10.743448 31973 tablet.cc:2369] T 0af9582263b842de9f78784ce62cb716 P 2d50f33557944bb480ea32199d9ee4f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:10.746328 31927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af9582263b842de9f78784ce62cb716" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:10.746551 31927 raft_consensus.cc:2454] T 0af9582263b842de9f78784ce62cb716 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:10.747014 31733 leader_election.cc:302] T 0af9582263b842de9f78784ce62cb716 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:10.747339 32524 raft_consensus.cc:2781] T 0af9582263b842de9f78784ce62cb716 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:10.747469 32524 raft_consensus.cc:482] T 0af9582263b842de9f78784ce62cb716 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:10.747566 32524 raft_consensus.cc:3037] T 0af9582263b842de9f78784ce62cb716 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:10.750105 32524 raft_consensus.cc:504] T 0af9582263b842de9f78784ce62cb716 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:10.750700 32524 leader_election.cc:288] T 0af9582263b842de9f78784ce62cb716 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Requested vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:10.750713 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af9582263b842de9f78784ce62cb716" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:10.751010 31927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af9582263b842de9f78784ce62cb716" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:10.751135 31536 raft_consensus.cc:2454] T 0af9582263b842de9f78784ce62cb716 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:10.751245 31927 raft_consensus.cc:3037] T 0af9582263b842de9f78784ce62cb716 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:10.751343 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af9582263b842de9f78784ce62cb716" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:26:10.751621 31535 raft_consensus.cc:3037] T 0af9582263b842de9f78784ce62cb716 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:10.753664 31927 raft_consensus.cc:2454] T 0af9582263b842de9f78784ce62cb716 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:10.753971 31733 leader_election.cc:302] T 0af9582263b842de9f78784ce62cb716 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:10.754135 31535 raft_consensus.cc:2454] T 0af9582263b842de9f78784ce62cb716 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:10.754227 32524 raft_consensus.cc:2781] T 0af9582263b842de9f78784ce62cb716 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:10.754599 32524 raft_consensus.cc:686] T 0af9582263b842de9f78784ce62cb716 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Becoming Leader. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Running, Role: LEADER
I20240502 21:26:10.755015 32524 consensus_queue.cc:237] T 0af9582263b842de9f78784ce62cb716 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:10.758652 31393 catalog_manager.cc:5484] T 0af9582263b842de9f78784ce62cb716 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: term changed from 0 to 1, leader changed from <none> to 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131). New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } }
W20240502 21:26:10.762562 31582 tablet.cc:2369] T 0af9582263b842de9f78784ce62cb716 P a44f41b0b6644fb6aa29476aed8c964f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:10.797757 32530 raft_consensus.cc:482] T 174d4efa03b5450ebe34170e9279a390 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:10.797909 32530 raft_consensus.cc:504] T 174d4efa03b5450ebe34170e9279a390 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:10.799259 32530 leader_election.cc:288] T 174d4efa03b5450ebe34170e9279a390 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:10.804957 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174d4efa03b5450ebe34170e9279a390" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:10.805141 31535 raft_consensus.cc:2454] T 174d4efa03b5450ebe34170e9279a390 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:26:10.805055 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174d4efa03b5450ebe34170e9279a390" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:10.805414 31797 raft_consensus.cc:2454] T 174d4efa03b5450ebe34170e9279a390 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:26:10.805487 31603 leader_election.cc:302] T 174d4efa03b5450ebe34170e9279a390 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:10.805817 32530 raft_consensus.cc:2781] T 174d4efa03b5450ebe34170e9279a390 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:10.805933 32530 raft_consensus.cc:482] T 174d4efa03b5450ebe34170e9279a390 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:10.806020 32530 raft_consensus.cc:3037] T 174d4efa03b5450ebe34170e9279a390 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:10.807701 32530 raft_consensus.cc:504] T 174d4efa03b5450ebe34170e9279a390 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:10.808199 32530 leader_election.cc:288] T 174d4efa03b5450ebe34170e9279a390 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Requested vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:10.808504 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174d4efa03b5450ebe34170e9279a390" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:10.808643 31797 raft_consensus.cc:3037] T 174d4efa03b5450ebe34170e9279a390 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:10.808593 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174d4efa03b5450ebe34170e9279a390" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:26:10.808773 31535 raft_consensus.cc:3037] T 174d4efa03b5450ebe34170e9279a390 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:10.810353 31535 raft_consensus.cc:2454] T 174d4efa03b5450ebe34170e9279a390 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:10.810346 31797 raft_consensus.cc:2454] T 174d4efa03b5450ebe34170e9279a390 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:10.810704 31603 leader_election.cc:302] T 174d4efa03b5450ebe34170e9279a390 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:10.810987 32530 raft_consensus.cc:2781] T 174d4efa03b5450ebe34170e9279a390 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:10.811399 32530 raft_consensus.cc:686] T 174d4efa03b5450ebe34170e9279a390 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Becoming Leader. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Running, Role: LEADER
I20240502 21:26:10.811863 32530 consensus_queue.cc:237] T 174d4efa03b5450ebe34170e9279a390 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:10.815703 31393 catalog_manager.cc:5484] T 174d4efa03b5450ebe34170e9279a390 P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130). New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } }
W20240502 21:26:10.826566 31843 tablet.cc:2369] T 0af9582263b842de9f78784ce62cb716 P 08d7a0680d244b55b93d17f9ab57b84c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 21:26:11.126724 32549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:11.126998 32549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 21:26:11.132107 32534 consensus_queue.cc:1035] T 9d89305df0454659b691ad23a0ff3392 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:11.134835 32549 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:26:11.139681 32556 consensus_queue.cc:1035] T 9d89305df0454659b691ad23a0ff3392 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240502 21:26:11.211319 32530 consensus_queue.cc:1035] T 174d4efa03b5450ebe34170e9279a390 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:11.225713 32530 consensus_queue.cc:1035] T 174d4efa03b5450ebe34170e9279a390 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:11.327783 32524 consensus_queue.cc:1035] T 0af9582263b842de9f78784ce62cb716 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:11.345712 31927 raft_consensus.cc:1261] T 0af9582263b842de9f78784ce62cb716 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:11.360240 32524 consensus_queue.cc:1035] T 0af9582263b842de9f78784ce62cb716 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:11.368387 32586 mvcc.cc:205] Tried to move back new op lower bound from 7023350461699403776 to 7023350459415859200. Current Snapshot: MvccSnapshot[applied={T|T < 7023350461699403776}]
I20240502 21:26:11.418566 32587 mvcc.cc:205] Tried to move back new op lower bound from 7023350461699403776 to 7023350459415859200. Current Snapshot: MvccSnapshot[applied={T|T < 7023350461699403776}]
INSERT report
rows total: 2000
time total: 262.285 ms
time per row: 0.131143 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 21:26:12.383383 31392 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:12.391363 31646 tablet_service.cc:1467] Processing CreateTablet for tablet b407c2c9b3a14ce3ada52b7856c6ecee (DEFAULT_TABLE table=rebalance_test_table_1 [id=08ebfd17dee84efd8a264a88ead3fa46]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:12.391552 31776 tablet_service.cc:1467] Processing CreateTablet for tablet b407c2c9b3a14ce3ada52b7856c6ecee (DEFAULT_TABLE table=rebalance_test_table_1 [id=08ebfd17dee84efd8a264a88ead3fa46]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:12.391837 31646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b407c2c9b3a14ce3ada52b7856c6ecee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:12.391985 31776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b407c2c9b3a14ce3ada52b7856c6ecee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:12.392110 31777 tablet_service.cc:1467] Processing CreateTablet for tablet 5a54d2aad63c4f3182992f00b8564acf (DEFAULT_TABLE table=rebalance_test_table_1 [id=08ebfd17dee84efd8a264a88ead3fa46]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:12.392472 31777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a54d2aad63c4f3182992f00b8564acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:12.393230 31906 tablet_service.cc:1467] Processing CreateTablet for tablet b407c2c9b3a14ce3ada52b7856c6ecee (DEFAULT_TABLE table=rebalance_test_table_1 [id=08ebfd17dee84efd8a264a88ead3fa46]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:12.393319 31516 tablet_service.cc:1467] Processing CreateTablet for tablet 62e70b8e69c94d76b23c655a3f5de5bd (DEFAULT_TABLE table=rebalance_test_table_1 [id=08ebfd17dee84efd8a264a88ead3fa46]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:12.393613 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b407c2c9b3a14ce3ada52b7856c6ecee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:12.393733 31516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62e70b8e69c94d76b23c655a3f5de5bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:12.396749 31907 tablet_service.cc:1467] Processing CreateTablet for tablet 5a54d2aad63c4f3182992f00b8564acf (DEFAULT_TABLE table=rebalance_test_table_1 [id=08ebfd17dee84efd8a264a88ead3fa46]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:12.397080 31907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a54d2aad63c4f3182992f00b8564acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:12.399910 31775 tablet_service.cc:1467] Processing CreateTablet for tablet 62e70b8e69c94d76b23c655a3f5de5bd (DEFAULT_TABLE table=rebalance_test_table_1 [id=08ebfd17dee84efd8a264a88ead3fa46]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:12.400231 31775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62e70b8e69c94d76b23c655a3f5de5bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:12.405508 32618 tablet_bootstrap.cc:492] T 62e70b8e69c94d76b23c655a3f5de5bd P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:12.405463 31647 tablet_service.cc:1467] Processing CreateTablet for tablet 5a54d2aad63c4f3182992f00b8564acf (DEFAULT_TABLE table=rebalance_test_table_1 [id=08ebfd17dee84efd8a264a88ead3fa46]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:12.405618 31645 tablet_service.cc:1467] Processing CreateTablet for tablet 62e70b8e69c94d76b23c655a3f5de5bd (DEFAULT_TABLE table=rebalance_test_table_1 [id=08ebfd17dee84efd8a264a88ead3fa46]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:12.405907 31647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a54d2aad63c4f3182992f00b8564acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:12.406477 31645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62e70b8e69c94d76b23c655a3f5de5bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:12.407503 32618 tablet_bootstrap.cc:654] T 62e70b8e69c94d76b23c655a3f5de5bd P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:12.411844 32619 tablet_bootstrap.cc:492] T 5a54d2aad63c4f3182992f00b8564acf P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:12.414052 32619 tablet_bootstrap.cc:654] T 5a54d2aad63c4f3182992f00b8564acf P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:12.415338 32618 tablet_bootstrap.cc:492] T 62e70b8e69c94d76b23c655a3f5de5bd P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:12.415565 32618 ts_tablet_manager.cc:1367] T 62e70b8e69c94d76b23c655a3f5de5bd P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20240502 21:26:12.416323 32618 raft_consensus.cc:348] T 62e70b8e69c94d76b23c655a3f5de5bd P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:12.416669 32618 raft_consensus.cc:374] T 62e70b8e69c94d76b23c655a3f5de5bd P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:12.416793 32618 raft_consensus.cc:729] T 62e70b8e69c94d76b23c655a3f5de5bd P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:12.417093 32618 consensus_queue.cc:260] T 62e70b8e69c94d76b23c655a3f5de5bd P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:12.417769 32618 ts_tablet_manager.cc:1398] T 62e70b8e69c94d76b23c655a3f5de5bd P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:12.428370 32622 tablet_bootstrap.cc:492] T 62e70b8e69c94d76b23c655a3f5de5bd P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:12.428654 32617 tablet_bootstrap.cc:492] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:12.429056 32619 tablet_bootstrap.cc:492] T 5a54d2aad63c4f3182992f00b8564acf P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:12.429200 32619 ts_tablet_manager.cc:1367] T 5a54d2aad63c4f3182992f00b8564acf P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.010s
I20240502 21:26:12.429747 32619 raft_consensus.cc:348] T 5a54d2aad63c4f3182992f00b8564acf P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:12.429890 32619 raft_consensus.cc:374] T 5a54d2aad63c4f3182992f00b8564acf P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:12.429903 32622 tablet_bootstrap.cc:654] T 62e70b8e69c94d76b23c655a3f5de5bd P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:12.430004 32619 raft_consensus.cc:729] T 5a54d2aad63c4f3182992f00b8564acf P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:12.430166 32617 tablet_bootstrap.cc:654] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:12.430219 32619 consensus_queue.cc:260] T 5a54d2aad63c4f3182992f00b8564acf P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:12.430994 32619 ts_tablet_manager.cc:1398] T 5a54d2aad63c4f3182992f00b8564acf P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:12.431296 32619 tablet_bootstrap.cc:492] T b407c2c9b3a14ce3ada52b7856c6ecee P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:12.432806 32619 tablet_bootstrap.cc:654] T b407c2c9b3a14ce3ada52b7856c6ecee P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:12.435864 32617 tablet_bootstrap.cc:492] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:12.436040 32622 tablet_bootstrap.cc:492] T 62e70b8e69c94d76b23c655a3f5de5bd P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:12.436033 32617 ts_tablet_manager.cc:1367] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20240502 21:26:12.436161 32622 ts_tablet_manager.cc:1367] T 62e70b8e69c94d76b23c655a3f5de5bd P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 21:26:12.436800 32617 raft_consensus.cc:348] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:12.436949 32619 tablet_bootstrap.cc:492] T b407c2c9b3a14ce3ada52b7856c6ecee P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:12.436889 32622 raft_consensus.cc:348] T 62e70b8e69c94d76b23c655a3f5de5bd P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:12.437041 32617 raft_consensus.cc:374] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:12.437127 32619 ts_tablet_manager.cc:1367] T b407c2c9b3a14ce3ada52b7856c6ecee P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 21:26:12.437130 32622 raft_consensus.cc:374] T 62e70b8e69c94d76b23c655a3f5de5bd P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:12.437222 32617 raft_consensus.cc:729] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:12.437227 32622 raft_consensus.cc:729] T 62e70b8e69c94d76b23c655a3f5de5bd P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:12.437439 32622 consensus_queue.cc:260] T 62e70b8e69c94d76b23c655a3f5de5bd P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:12.437465 32617 consensus_queue.cc:260] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:12.437851 32622 ts_tablet_manager.cc:1398] T 62e70b8e69c94d76b23c655a3f5de5bd P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:26:12.437976 32617 ts_tablet_manager.cc:1398] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:12.438104 32622 tablet_bootstrap.cc:492] T 5a54d2aad63c4f3182992f00b8564acf P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:12.438274 32617 tablet_bootstrap.cc:492] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:12.438620 32619 raft_consensus.cc:348] T b407c2c9b3a14ce3ada52b7856c6ecee P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:12.438853 32619 raft_consensus.cc:374] T b407c2c9b3a14ce3ada52b7856c6ecee P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:12.438946 32619 raft_consensus.cc:729] T b407c2c9b3a14ce3ada52b7856c6ecee P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:12.439178 32619 consensus_queue.cc:260] T b407c2c9b3a14ce3ada52b7856c6ecee P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:12.439499 32622 tablet_bootstrap.cc:654] T 5a54d2aad63c4f3182992f00b8564acf P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:12.439674 32617 tablet_bootstrap.cc:654] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:12.439747 32619 ts_tablet_manager.cc:1398] T b407c2c9b3a14ce3ada52b7856c6ecee P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:12.441443 32617 tablet_bootstrap.cc:492] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:12.441640 32617 ts_tablet_manager.cc:1367] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.002s
I20240502 21:26:12.442364 32617 raft_consensus.cc:348] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:12.442595 32617 raft_consensus.cc:374] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:12.442688 32617 raft_consensus.cc:729] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:12.442914 32617 consensus_queue.cc:260] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:12.443403 32617 ts_tablet_manager.cc:1398] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:26:12.443655 32617 tablet_bootstrap.cc:492] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:12.444862 32617 tablet_bootstrap.cc:654] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:12.448738 32617 tablet_bootstrap.cc:492] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:12.448848 32622 tablet_bootstrap.cc:492] T 5a54d2aad63c4f3182992f00b8564acf P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:12.448876 32617 ts_tablet_manager.cc:1367] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240502 21:26:12.449005 32622 ts_tablet_manager.cc:1367] T 5a54d2aad63c4f3182992f00b8564acf P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.006s
I20240502 21:26:12.449412 32617 raft_consensus.cc:348] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:12.449568 32617 raft_consensus.cc:374] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:12.449514 32622 raft_consensus.cc:348] T 5a54d2aad63c4f3182992f00b8564acf P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:12.449647 32617 raft_consensus.cc:729] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:12.449703 32622 raft_consensus.cc:374] T 5a54d2aad63c4f3182992f00b8564acf P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:12.449779 32622 raft_consensus.cc:729] T 5a54d2aad63c4f3182992f00b8564acf P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:12.449833 32617 consensus_queue.cc:260] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:12.449978 32622 consensus_queue.cc:260] T 5a54d2aad63c4f3182992f00b8564acf P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:12.450310 32617 ts_tablet_manager.cc:1398] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:26:12.451005 32622 ts_tablet_manager.cc:1398] T 5a54d2aad63c4f3182992f00b8564acf P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:12.451232 32622 tablet_bootstrap.cc:492] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:12.452523 32622 tablet_bootstrap.cc:654] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:12.454082 32622 tablet_bootstrap.cc:492] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:12.454198 32622 ts_tablet_manager.cc:1367] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:26:12.454834 32622 raft_consensus.cc:348] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:12.454962 32622 raft_consensus.cc:374] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:12.455052 32622 raft_consensus.cc:729] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:12.455248 32622 consensus_queue.cc:260] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:12.455670 32622 ts_tablet_manager.cc:1398] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 21:26:12.494341 32612 raft_consensus.cc:482] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:12.494499 32612 raft_consensus.cc:504] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:12.495113 32612 leader_election.cc:288] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:12.496223 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e70b8e69c94d76b23c655a3f5de5bd" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:12.496519 31536 raft_consensus.cc:2454] T 62e70b8e69c94d76b23c655a3f5de5bd P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:12.496958 31733 leader_election.cc:302] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:12.497182 32612 raft_consensus.cc:2781] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:12.497287 32612 raft_consensus.cc:482] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:12.497368 32612 raft_consensus.cc:3037] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:12.499182 32612 raft_consensus.cc:504] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:12.499670 32612 leader_election.cc:288] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Requested vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:12.500201 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e70b8e69c94d76b23c655a3f5de5bd" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:26:12.500413 31536 raft_consensus.cc:3037] T 62e70b8e69c94d76b23c655a3f5de5bd P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:12.500839 31667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e70b8e69c94d76b23c655a3f5de5bd" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:12.501075 31667 raft_consensus.cc:2454] T 62e70b8e69c94d76b23c655a3f5de5bd P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:12.501009 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e70b8e69c94d76b23c655a3f5de5bd" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:12.501273 31666 raft_consensus.cc:3037] T 62e70b8e69c94d76b23c655a3f5de5bd P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:12.503098 31536 raft_consensus.cc:2454] T 62e70b8e69c94d76b23c655a3f5de5bd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:12.503434 31733 leader_election.cc:302] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:12.503726 32612 raft_consensus.cc:2781] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:12.503875 31666 raft_consensus.cc:2454] T 62e70b8e69c94d76b23c655a3f5de5bd P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:12.504093 32612 raft_consensus.cc:686] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Becoming Leader. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Running, Role: LEADER
I20240502 21:26:12.504374 32612 consensus_queue.cc:237] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:12.506657 31387 catalog_manager.cc:5484] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: term changed from 0 to 1, leader changed from <none> to 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131). New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:12.560112 32612 raft_consensus.cc:482] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:12.560299 32612 raft_consensus.cc:504] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:12.560918 32612 leader_election.cc:288] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:12.561393 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a54d2aad63c4f3182992f00b8564acf" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:12.561625 31927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a54d2aad63c4f3182992f00b8564acf" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:12.561669 31666 raft_consensus.cc:2454] T 5a54d2aad63c4f3182992f00b8564acf P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:12.561841 31927 raft_consensus.cc:2454] T 5a54d2aad63c4f3182992f00b8564acf P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:12.562026 31733 leader_election.cc:302] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 08d7a0680d244b55b93d17f9ab57b84c; no voters:
I20240502 21:26:12.562399 32612 raft_consensus.cc:2781] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:12.562502 32612 raft_consensus.cc:482] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:12.562616 32612 raft_consensus.cc:3037] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:12.564424 32612 raft_consensus.cc:504] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:12.564893 32612 leader_election.cc:288] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Requested vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:12.565251 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a54d2aad63c4f3182992f00b8564acf" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:12.565305 31927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a54d2aad63c4f3182992f00b8564acf" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:12.565479 31666 raft_consensus.cc:3037] T 5a54d2aad63c4f3182992f00b8564acf P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:12.565513 31927 raft_consensus.cc:3037] T 5a54d2aad63c4f3182992f00b8564acf P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:12.568099 31666 raft_consensus.cc:2454] T 5a54d2aad63c4f3182992f00b8564acf P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:12.568112 31927 raft_consensus.cc:2454] T 5a54d2aad63c4f3182992f00b8564acf P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:12.568538 31733 leader_election.cc:302] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 08d7a0680d244b55b93d17f9ab57b84c; no voters:
I20240502 21:26:12.568775 32612 raft_consensus.cc:2781] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:12.568936 32612 raft_consensus.cc:686] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Becoming Leader. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Running, Role: LEADER
I20240502 21:26:12.569175 32612 consensus_queue.cc:237] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:12.571480 31387 catalog_manager.cc:5484] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: term changed from 0 to 1, leader changed from <none> to 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131). New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:12.757649 32612 raft_consensus.cc:482] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:12.757828 32612 raft_consensus.cc:504] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:12.758491 32612 leader_election.cc:288] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:12.758935 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b407c2c9b3a14ce3ada52b7856c6ecee" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:12.758950 31927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b407c2c9b3a14ce3ada52b7856c6ecee" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:12.759161 31927 raft_consensus.cc:2454] T b407c2c9b3a14ce3ada52b7856c6ecee P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:12.759160 31666 raft_consensus.cc:2454] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:12.759559 31733 leader_election.cc:302] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 08d7a0680d244b55b93d17f9ab57b84c; no voters:
I20240502 21:26:12.759812 32612 raft_consensus.cc:2781] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:12.759914 32612 raft_consensus.cc:482] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:12.759999 32612 raft_consensus.cc:3037] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:12.762022 32612 raft_consensus.cc:504] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:12.762475 32612 leader_election.cc:288] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Requested vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:12.762890 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b407c2c9b3a14ce3ada52b7856c6ecee" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:12.762915 31927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b407c2c9b3a14ce3ada52b7856c6ecee" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:12.763058 31927 raft_consensus.cc:3037] T b407c2c9b3a14ce3ada52b7856c6ecee P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:12.763042 31666 raft_consensus.cc:3037] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:12.764961 31927 raft_consensus.cc:2454] T b407c2c9b3a14ce3ada52b7856c6ecee P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:12.765061 31666 raft_consensus.cc:2454] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:12.765368 31733 leader_election.cc:302] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:12.765617 32612 raft_consensus.cc:2781] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:12.765748 32612 raft_consensus.cc:686] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Becoming Leader. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Running, Role: LEADER
I20240502 21:26:12.766016 32612 consensus_queue.cc:237] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:12.768301 31387 catalog_manager.cc:5484] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: term changed from 0 to 1, leader changed from <none> to 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131). New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:12.917418 32612 consensus_queue.cc:1035] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:12.923214 32612 consensus_queue.cc:1035] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:12.970538 32633 consensus_queue.cc:1035] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:12.974613 32633 consensus_queue.cc:1035] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:13.165596 32633 consensus_queue.cc:1035] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:13.171347 32633 consensus_queue.cc:1035] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:13.184772 32637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:13.185042 32637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:13.192232 32637 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 158.075 ms
time per row: 0.0790375 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 21:26:14.346626 31387 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:14.355007 31516 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7bd0bcc3114425aa0dd1616488d612 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ed5c1c20aa174a8e845358f36f4420a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:14.355460 31516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7bd0bcc3114425aa0dd1616488d612. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:14.356070 31907 tablet_service.cc:1467] Processing CreateTablet for tablet 721d6ac1f9294fb4a7280c07bce3bf62 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ed5c1c20aa174a8e845358f36f4420a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:14.356467 31907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 721d6ac1f9294fb4a7280c07bce3bf62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:14.356626 31645 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7bd0bcc3114425aa0dd1616488d612 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ed5c1c20aa174a8e845358f36f4420a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:14.357014 31645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7bd0bcc3114425aa0dd1616488d612. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:14.359278 31646 tablet_service.cc:1467] Processing CreateTablet for tablet 0057ca44a6c34cecb810133bd0120615 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ed5c1c20aa174a8e845358f36f4420a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:14.359467 31906 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7bd0bcc3114425aa0dd1616488d612 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ed5c1c20aa174a8e845358f36f4420a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:14.359611 31646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0057ca44a6c34cecb810133bd0120615. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:14.359818 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7bd0bcc3114425aa0dd1616488d612. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:14.362291 31776 tablet_service.cc:1467] Processing CreateTablet for tablet 721d6ac1f9294fb4a7280c07bce3bf62 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ed5c1c20aa174a8e845358f36f4420a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:14.362679 31776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 721d6ac1f9294fb4a7280c07bce3bf62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:14.363013 31905 tablet_service.cc:1467] Processing CreateTablet for tablet 0057ca44a6c34cecb810133bd0120615 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ed5c1c20aa174a8e845358f36f4420a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:14.363337 31905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0057ca44a6c34cecb810133bd0120615. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:14.363679 31515 tablet_service.cc:1467] Processing CreateTablet for tablet 0057ca44a6c34cecb810133bd0120615 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ed5c1c20aa174a8e845358f36f4420a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:14.363996 31515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0057ca44a6c34cecb810133bd0120615. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:14.364740 31514 tablet_service.cc:1467] Processing CreateTablet for tablet 721d6ac1f9294fb4a7280c07bce3bf62 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ed5c1c20aa174a8e845358f36f4420a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:14.365029 31514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 721d6ac1f9294fb4a7280c07bce3bf62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:14.371246 32702 tablet_bootstrap.cc:492] T 0057ca44a6c34cecb810133bd0120615 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:14.371594 32701 tablet_bootstrap.cc:492] T 7d7bd0bcc3114425aa0dd1616488d612 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:14.372885 32702 tablet_bootstrap.cc:654] T 0057ca44a6c34cecb810133bd0120615 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:14.372996 32701 tablet_bootstrap.cc:654] T 7d7bd0bcc3114425aa0dd1616488d612 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:14.374680 32702 tablet_bootstrap.cc:492] T 0057ca44a6c34cecb810133bd0120615 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:14.374823 32702 ts_tablet_manager.cc:1367] T 0057ca44a6c34cecb810133bd0120615 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:26:14.375615 32702 raft_consensus.cc:348] T 0057ca44a6c34cecb810133bd0120615 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:14.375802 32702 raft_consensus.cc:374] T 0057ca44a6c34cecb810133bd0120615 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:14.375888 32702 raft_consensus.cc:729] T 0057ca44a6c34cecb810133bd0120615 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:14.376080 32702 consensus_queue.cc:260] T 0057ca44a6c34cecb810133bd0120615 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:14.376626 32702 ts_tablet_manager.cc:1398] T 0057ca44a6c34cecb810133bd0120615 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:14.376896 32702 tablet_bootstrap.cc:492] T 7d7bd0bcc3114425aa0dd1616488d612 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:14.378288 32702 tablet_bootstrap.cc:654] T 7d7bd0bcc3114425aa0dd1616488d612 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:14.380625 32701 tablet_bootstrap.cc:492] T 7d7bd0bcc3114425aa0dd1616488d612 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:14.380776 32701 ts_tablet_manager.cc:1367] T 7d7bd0bcc3114425aa0dd1616488d612 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240502 21:26:14.381731 32701 raft_consensus.cc:348] T 7d7bd0bcc3114425aa0dd1616488d612 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:14.381932 32701 raft_consensus.cc:374] T 7d7bd0bcc3114425aa0dd1616488d612 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:14.382057 32701 raft_consensus.cc:729] T 7d7bd0bcc3114425aa0dd1616488d612 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:14.382323 32701 consensus_queue.cc:260] T 7d7bd0bcc3114425aa0dd1616488d612 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:14.383457 32701 ts_tablet_manager.cc:1398] T 7d7bd0bcc3114425aa0dd1616488d612 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:14.383585 32702 tablet_bootstrap.cc:492] T 7d7bd0bcc3114425aa0dd1616488d612 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:14.383723 32702 ts_tablet_manager.cc:1367] T 7d7bd0bcc3114425aa0dd1616488d612 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20240502 21:26:14.384351 32702 raft_consensus.cc:348] T 7d7bd0bcc3114425aa0dd1616488d612 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:14.384572 32702 raft_consensus.cc:374] T 7d7bd0bcc3114425aa0dd1616488d612 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:14.384642 32702 raft_consensus.cc:729] T 7d7bd0bcc3114425aa0dd1616488d612 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:14.384696 32701 tablet_bootstrap.cc:492] T 0057ca44a6c34cecb810133bd0120615 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:14.384958 32702 consensus_queue.cc:260] T 7d7bd0bcc3114425aa0dd1616488d612 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:14.385620 32702 ts_tablet_manager.cc:1398] T 7d7bd0bcc3114425aa0dd1616488d612 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:14.386087 32703 tablet_bootstrap.cc:492] T 0057ca44a6c34cecb810133bd0120615 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:14.386353 32701 tablet_bootstrap.cc:654] T 0057ca44a6c34cecb810133bd0120615 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:14.387746 32703 tablet_bootstrap.cc:654] T 0057ca44a6c34cecb810133bd0120615 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:14.389762 32706 tablet_bootstrap.cc:492] T 721d6ac1f9294fb4a7280c07bce3bf62 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:14.389989 32703 tablet_bootstrap.cc:492] T 0057ca44a6c34cecb810133bd0120615 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:14.390185 32703 ts_tablet_manager.cc:1367] T 0057ca44a6c34cecb810133bd0120615 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 21:26:14.390532 32701 tablet_bootstrap.cc:492] T 0057ca44a6c34cecb810133bd0120615 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:14.390667 32701 ts_tablet_manager.cc:1367] T 0057ca44a6c34cecb810133bd0120615 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20240502 21:26:14.390890 32703 raft_consensus.cc:348] T 0057ca44a6c34cecb810133bd0120615 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:14.391041 32703 raft_consensus.cc:374] T 0057ca44a6c34cecb810133bd0120615 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:14.391139 32703 raft_consensus.cc:729] T 0057ca44a6c34cecb810133bd0120615 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:14.391201 32706 tablet_bootstrap.cc:654] T 721d6ac1f9294fb4a7280c07bce3bf62 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:14.391352 32703 consensus_queue.cc:260] T 0057ca44a6c34cecb810133bd0120615 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:14.391427 32701 raft_consensus.cc:348] T 0057ca44a6c34cecb810133bd0120615 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:14.391680 32701 raft_consensus.cc:374] T 0057ca44a6c34cecb810133bd0120615 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:14.391782 32701 raft_consensus.cc:729] T 0057ca44a6c34cecb810133bd0120615 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:14.391929 32703 ts_tablet_manager.cc:1398] T 0057ca44a6c34cecb810133bd0120615 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:14.392071 32701 consensus_queue.cc:260] T 0057ca44a6c34cecb810133bd0120615 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:14.392235 32703 tablet_bootstrap.cc:492] T 721d6ac1f9294fb4a7280c07bce3bf62 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:14.393038 32701 ts_tablet_manager.cc:1398] T 0057ca44a6c34cecb810133bd0120615 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:14.393360 32701 tablet_bootstrap.cc:492] T 721d6ac1f9294fb4a7280c07bce3bf62 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:14.393646 32703 tablet_bootstrap.cc:654] T 721d6ac1f9294fb4a7280c07bce3bf62 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:14.393805 32706 tablet_bootstrap.cc:492] T 721d6ac1f9294fb4a7280c07bce3bf62 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:14.393992 32706 ts_tablet_manager.cc:1367] T 721d6ac1f9294fb4a7280c07bce3bf62 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:26:14.394845 32706 raft_consensus.cc:348] T 721d6ac1f9294fb4a7280c07bce3bf62 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:14.395260 32701 tablet_bootstrap.cc:654] T 721d6ac1f9294fb4a7280c07bce3bf62 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:14.395262 32706 raft_consensus.cc:374] T 721d6ac1f9294fb4a7280c07bce3bf62 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:14.395352 32706 raft_consensus.cc:729] T 721d6ac1f9294fb4a7280c07bce3bf62 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:14.395633 32706 consensus_queue.cc:260] T 721d6ac1f9294fb4a7280c07bce3bf62 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:14.397176 32706 ts_tablet_manager.cc:1398] T 721d6ac1f9294fb4a7280c07bce3bf62 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:26:14.398265 32701 tablet_bootstrap.cc:492] T 721d6ac1f9294fb4a7280c07bce3bf62 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:14.398463 32701 ts_tablet_manager.cc:1367] T 721d6ac1f9294fb4a7280c07bce3bf62 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240502 21:26:14.399147 32701 raft_consensus.cc:348] T 721d6ac1f9294fb4a7280c07bce3bf62 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:14.399377 32701 raft_consensus.cc:374] T 721d6ac1f9294fb4a7280c07bce3bf62 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:14.399466 32701 raft_consensus.cc:729] T 721d6ac1f9294fb4a7280c07bce3bf62 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:14.399669 32701 consensus_queue.cc:260] T 721d6ac1f9294fb4a7280c07bce3bf62 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:14.400122 32703 tablet_bootstrap.cc:492] T 721d6ac1f9294fb4a7280c07bce3bf62 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:14.400291 32703 ts_tablet_manager.cc:1367] T 721d6ac1f9294fb4a7280c07bce3bf62 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240502 21:26:14.400743 32701 ts_tablet_manager.cc:1398] T 721d6ac1f9294fb4a7280c07bce3bf62 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:14.400899 32703 raft_consensus.cc:348] T 721d6ac1f9294fb4a7280c07bce3bf62 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:14.401118 32703 raft_consensus.cc:374] T 721d6ac1f9294fb4a7280c07bce3bf62 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:14.401223 32703 raft_consensus.cc:729] T 721d6ac1f9294fb4a7280c07bce3bf62 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:14.401463 32703 consensus_queue.cc:260] T 721d6ac1f9294fb4a7280c07bce3bf62 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:14.401993 32703 ts_tablet_manager.cc:1398] T 721d6ac1f9294fb4a7280c07bce3bf62 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:14.402305 32703 tablet_bootstrap.cc:492] T 7d7bd0bcc3114425aa0dd1616488d612 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:14.403769 32703 tablet_bootstrap.cc:654] T 7d7bd0bcc3114425aa0dd1616488d612 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:14.405506 32703 tablet_bootstrap.cc:492] T 7d7bd0bcc3114425aa0dd1616488d612 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:14.405685 32703 ts_tablet_manager.cc:1367] T 7d7bd0bcc3114425aa0dd1616488d612 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20240502 21:26:14.406275 32703 raft_consensus.cc:348] T 7d7bd0bcc3114425aa0dd1616488d612 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:14.406414 32703 raft_consensus.cc:374] T 7d7bd0bcc3114425aa0dd1616488d612 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:14.406497 32703 raft_consensus.cc:729] T 7d7bd0bcc3114425aa0dd1616488d612 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:14.406703 32703 consensus_queue.cc:260] T 7d7bd0bcc3114425aa0dd1616488d612 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:14.407038 32703 ts_tablet_manager.cc:1398] T 7d7bd0bcc3114425aa0dd1616488d612 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:26:14.436964 32708 raft_consensus.cc:482] T 721d6ac1f9294fb4a7280c07bce3bf62 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:14.437099 32708 raft_consensus.cc:504] T 721d6ac1f9294fb4a7280c07bce3bf62 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:14.438477 32708 leader_election.cc:288] T 721d6ac1f9294fb4a7280c07bce3bf62 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:14.451938 32534 raft_consensus.cc:482] T 7d7bd0bcc3114425aa0dd1616488d612 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:14.452126 32534 raft_consensus.cc:504] T 7d7bd0bcc3114425aa0dd1616488d612 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:14.452769 32534 leader_election.cc:288] T 7d7bd0bcc3114425aa0dd1616488d612 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:14.453187 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7bd0bcc3114425aa0dd1616488d612" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:14.453398 31926 raft_consensus.cc:2454] T 7d7bd0bcc3114425aa0dd1616488d612 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:14.453770 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7bd0bcc3114425aa0dd1616488d612" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:14.453971 31665 raft_consensus.cc:2454] T 7d7bd0bcc3114425aa0dd1616488d612 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:14.454416 31472 leader_election.cc:302] T 7d7bd0bcc3114425aa0dd1616488d612 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:14.454632 32534 raft_consensus.cc:2781] T 7d7bd0bcc3114425aa0dd1616488d612 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:14.454741 32534 raft_consensus.cc:482] T 7d7bd0bcc3114425aa0dd1616488d612 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:14.454823 32534 raft_consensus.cc:3037] T 7d7bd0bcc3114425aa0dd1616488d612 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:14.456912 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721d6ac1f9294fb4a7280c07bce3bf62" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:14.457129 31536 raft_consensus.cc:2454] T 721d6ac1f9294fb4a7280c07bce3bf62 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:14.457420 32534 raft_consensus.cc:504] T 7d7bd0bcc3114425aa0dd1616488d612 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:14.457458 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721d6ac1f9294fb4a7280c07bce3bf62" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:14.457610 31863 leader_election.cc:302] T 721d6ac1f9294fb4a7280c07bce3bf62 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:14.457695 31797 raft_consensus.cc:2454] T 721d6ac1f9294fb4a7280c07bce3bf62 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:14.457945 32708 raft_consensus.cc:2781] T 721d6ac1f9294fb4a7280c07bce3bf62 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:14.458022 32534 leader_election.cc:288] T 7d7bd0bcc3114425aa0dd1616488d612 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Requested vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:14.458047 32708 raft_consensus.cc:482] T 721d6ac1f9294fb4a7280c07bce3bf62 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:14.458132 32708 raft_consensus.cc:3037] T 721d6ac1f9294fb4a7280c07bce3bf62 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:14.458313 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7bd0bcc3114425aa0dd1616488d612" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:14.458499 31665 raft_consensus.cc:3037] T 7d7bd0bcc3114425aa0dd1616488d612 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:14.458680 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7bd0bcc3114425aa0dd1616488d612" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:14.458873 31926 raft_consensus.cc:3037] T 7d7bd0bcc3114425aa0dd1616488d612 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:14.460139 32708 raft_consensus.cc:504] T 721d6ac1f9294fb4a7280c07bce3bf62 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:14.460729 32708 leader_election.cc:288] T 721d6ac1f9294fb4a7280c07bce3bf62 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Requested vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:14.461117 31665 raft_consensus.cc:2454] T 7d7bd0bcc3114425aa0dd1616488d612 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:14.461185 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721d6ac1f9294fb4a7280c07bce3bf62" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:26:14.461441 31536 raft_consensus.cc:3037] T 721d6ac1f9294fb4a7280c07bce3bf62 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:14.461578 31472 leader_election.cc:302] T 7d7bd0bcc3114425aa0dd1616488d612 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:14.461865 32534 raft_consensus.cc:2781] T 7d7bd0bcc3114425aa0dd1616488d612 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:14.461902 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721d6ac1f9294fb4a7280c07bce3bf62" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:14.462112 31797 raft_consensus.cc:3037] T 721d6ac1f9294fb4a7280c07bce3bf62 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:14.462324 32534 raft_consensus.cc:686] T 7d7bd0bcc3114425aa0dd1616488d612 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:26:14.462666 32534 consensus_queue.cc:237] T 7d7bd0bcc3114425aa0dd1616488d612 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:14.464067 31536 raft_consensus.cc:2454] T 721d6ac1f9294fb4a7280c07bce3bf62 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:26:14.464231 31797 raft_consensus.cc:2454] T 721d6ac1f9294fb4a7280c07bce3bf62 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:26:14.464350 31863 leader_election.cc:302] T 721d6ac1f9294fb4a7280c07bce3bf62 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:14.464651 32708 raft_consensus.cc:2781] T 721d6ac1f9294fb4a7280c07bce3bf62 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:14.465137 31926 raft_consensus.cc:2454] T 7d7bd0bcc3114425aa0dd1616488d612 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:14.465196 32708 raft_consensus.cc:686] T 721d6ac1f9294fb4a7280c07bce3bf62 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Running, Role: LEADER
I20240502 21:26:14.465818 32708 consensus_queue.cc:237] T 721d6ac1f9294fb4a7280c07bce3bf62 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:14.465981 31394 catalog_manager.cc:5484] T 7d7bd0bcc3114425aa0dd1616488d612 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 0 to 1, leader changed from <none> to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:14.469555 31387 catalog_manager.cc:5484] T 721d6ac1f9294fb4a7280c07bce3bf62 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132). New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:14.485983 32547 raft_consensus.cc:482] T 0057ca44a6c34cecb810133bd0120615 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:14.486138 32547 raft_consensus.cc:504] T 0057ca44a6c34cecb810133bd0120615 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:14.486814 32547 leader_election.cc:288] T 0057ca44a6c34cecb810133bd0120615 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:14.487190 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0057ca44a6c34cecb810133bd0120615" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:14.487401 31536 raft_consensus.cc:2454] T 0057ca44a6c34cecb810133bd0120615 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:26:14.487821 31603 leader_election.cc:302] T 0057ca44a6c34cecb810133bd0120615 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:14.488070 32547 raft_consensus.cc:2781] T 0057ca44a6c34cecb810133bd0120615 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:14.488185 32547 raft_consensus.cc:482] T 0057ca44a6c34cecb810133bd0120615 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:14.488279 32547 raft_consensus.cc:3037] T 0057ca44a6c34cecb810133bd0120615 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:14.490036 32547 raft_consensus.cc:504] T 0057ca44a6c34cecb810133bd0120615 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:14.490548 32547 leader_election.cc:288] T 0057ca44a6c34cecb810133bd0120615 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Requested vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:14.490862 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0057ca44a6c34cecb810133bd0120615" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:26:14.491006 31536 raft_consensus.cc:3037] T 0057ca44a6c34cecb810133bd0120615 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:14.492573 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0057ca44a6c34cecb810133bd0120615" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:14.492724 31927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0057ca44a6c34cecb810133bd0120615" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:14.492777 31926 raft_consensus.cc:2454] T 0057ca44a6c34cecb810133bd0120615 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:26:14.492903 31927 raft_consensus.cc:3037] T 0057ca44a6c34cecb810133bd0120615 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:14.492957 31536 raft_consensus.cc:2454] T 0057ca44a6c34cecb810133bd0120615 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:14.493306 31603 leader_election.cc:302] T 0057ca44a6c34cecb810133bd0120615 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:14.493561 32547 raft_consensus.cc:2781] T 0057ca44a6c34cecb810133bd0120615 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:14.493901 32547 raft_consensus.cc:686] T 0057ca44a6c34cecb810133bd0120615 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Becoming Leader. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Running, Role: LEADER
I20240502 21:26:14.494179 32547 consensus_queue.cc:237] T 0057ca44a6c34cecb810133bd0120615 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:14.494832 31927 raft_consensus.cc:2454] T 0057ca44a6c34cecb810133bd0120615 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:14.496311 31387 catalog_manager.cc:5484] T 0057ca44a6c34cecb810133bd0120615 P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130). New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } }
W20240502 21:26:14.840265 32727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:14.840664 32727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:14.848261 32727 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 21:26:14.964815 32708 consensus_queue.cc:1035] T 721d6ac1f9294fb4a7280c07bce3bf62 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:14.973415 31797 raft_consensus.cc:1261] T 721d6ac1f9294fb4a7280c07bce3bf62 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:14.974005 32708 consensus_queue.cc:1035] T 721d6ac1f9294fb4a7280c07bce3bf62 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:14.986286 32534 consensus_queue.cc:1035] T 7d7bd0bcc3114425aa0dd1616488d612 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:15.003223 31665 raft_consensus.cc:1261] T 7d7bd0bcc3114425aa0dd1616488d612 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:15.003842 32534 consensus_queue.cc:1035] T 7d7bd0bcc3114425aa0dd1616488d612 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:15.029259 31535 raft_consensus.cc:1261] T 0057ca44a6c34cecb810133bd0120615 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:15.030192 31926 raft_consensus.cc:1261] T 0057ca44a6c34cecb810133bd0120615 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:15.030167 32547 consensus_queue.cc:1035] T 0057ca44a6c34cecb810133bd0120615 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:15.030915 32547 consensus_queue.cc:1035] T 0057ca44a6c34cecb810133bd0120615 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:15.046108 32763 mvcc.cc:205] Tried to move back new op lower bound from 7023350476806696960 to 7023350474601402368. Current Snapshot: MvccSnapshot[applied={T|T < 7023350476806696960}]
INSERT report
rows total: 2000
time total: 274.939 ms
time per row: 0.13747 ms
I20240502 21:26:15.255041 31392 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:15.263664 31515 tablet_service.cc:1467] Processing CreateTablet for tablet 182b3f15c06c4c07ade0a8854d3ecafd (DEFAULT_TABLE table=rebalance_test_table_3 [id=59305f5894b34d0085a6cada5a676ca4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:15.264197 31515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182b3f15c06c4c07ade0a8854d3ecafd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:15.264588 31646 tablet_service.cc:1467] Processing CreateTablet for tablet 182b3f15c06c4c07ade0a8854d3ecafd (DEFAULT_TABLE table=rebalance_test_table_3 [id=59305f5894b34d0085a6cada5a676ca4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:15.264948 31646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182b3f15c06c4c07ade0a8854d3ecafd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:15.265412 31776 tablet_service.cc:1467] Processing CreateTablet for tablet 182b3f15c06c4c07ade0a8854d3ecafd (DEFAULT_TABLE table=rebalance_test_table_3 [id=59305f5894b34d0085a6cada5a676ca4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:15.265664 31777 tablet_service.cc:1467] Processing CreateTablet for tablet e0f96b56fb68439a97cf2aae4aa89b57 (DEFAULT_TABLE table=rebalance_test_table_3 [id=59305f5894b34d0085a6cada5a676ca4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:15.265818 31776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182b3f15c06c4c07ade0a8854d3ecafd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:15.266122 31777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f96b56fb68439a97cf2aae4aa89b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:15.269990 31775 tablet_service.cc:1467] Processing CreateTablet for tablet b20df8896a2b423cb53c2c1a2a1b98e2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=59305f5894b34d0085a6cada5a676ca4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:15.270360 31775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b20df8896a2b423cb53c2c1a2a1b98e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:15.275954 31906 tablet_service.cc:1467] Processing CreateTablet for tablet b20df8896a2b423cb53c2c1a2a1b98e2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=59305f5894b34d0085a6cada5a676ca4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:15.276278 31516 tablet_service.cc:1467] Processing CreateTablet for tablet b20df8896a2b423cb53c2c1a2a1b98e2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=59305f5894b34d0085a6cada5a676ca4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:15.276458 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b20df8896a2b423cb53c2c1a2a1b98e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:15.276774 31516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b20df8896a2b423cb53c2c1a2a1b98e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:15.279717 31907 tablet_service.cc:1467] Processing CreateTablet for tablet e0f96b56fb68439a97cf2aae4aa89b57 (DEFAULT_TABLE table=rebalance_test_table_3 [id=59305f5894b34d0085a6cada5a676ca4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:15.279773 31645 tablet_service.cc:1467] Processing CreateTablet for tablet e0f96b56fb68439a97cf2aae4aa89b57 (DEFAULT_TABLE table=rebalance_test_table_3 [id=59305f5894b34d0085a6cada5a676ca4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:15.280105 31907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f96b56fb68439a97cf2aae4aa89b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:15.280157 31645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f96b56fb68439a97cf2aae4aa89b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:15.288070 326 tablet_bootstrap.cc:492] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:15.290990 326 tablet_bootstrap.cc:654] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:15.302186 326 tablet_bootstrap.cc:492] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:15.302987 326 ts_tablet_manager.cc:1367] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20240502 21:26:15.304000 329 tablet_bootstrap.cc:492] T b20df8896a2b423cb53c2c1a2a1b98e2 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:15.304270 326 raft_consensus.cc:348] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:15.305915 326 raft_consensus.cc:374] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:15.306223 326 raft_consensus.cc:729] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:15.306542 329 tablet_bootstrap.cc:654] T b20df8896a2b423cb53c2c1a2a1b98e2 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:15.306890 326 consensus_queue.cc:260] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:15.308930 327 tablet_bootstrap.cc:492] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:15.309466 326 ts_tablet_manager.cc:1398] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 21:26:15.310276 326 tablet_bootstrap.cc:492] T 182b3f15c06c4c07ade0a8854d3ecafd P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:15.311023 327 tablet_bootstrap.cc:654] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:15.312234 326 tablet_bootstrap.cc:654] T 182b3f15c06c4c07ade0a8854d3ecafd P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:15.315505 325 tablet_bootstrap.cc:492] T 182b3f15c06c4c07ade0a8854d3ecafd P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:15.317857 325 tablet_bootstrap.cc:654] T 182b3f15c06c4c07ade0a8854d3ecafd P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:15.318114 327 tablet_bootstrap.cc:492] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:15.318285 327 ts_tablet_manager.cc:1367] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20240502 21:26:15.318981 329 tablet_bootstrap.cc:492] T b20df8896a2b423cb53c2c1a2a1b98e2 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:15.319108 327 raft_consensus.cc:348] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:15.319356 327 raft_consensus.cc:374] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:15.319483 327 raft_consensus.cc:729] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:15.319733 329 ts_tablet_manager.cc:1367] T b20df8896a2b423cb53c2c1a2a1b98e2 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20240502 21:26:15.319736 327 consensus_queue.cc:260] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:15.320297 327 ts_tablet_manager.cc:1398] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:26:15.320694 327 tablet_bootstrap.cc:492] T e0f96b56fb68439a97cf2aae4aa89b57 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:15.322170 327 tablet_bootstrap.cc:654] T e0f96b56fb68439a97cf2aae4aa89b57 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:15.326040 32765 raft_consensus.cc:482] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:15.326115 325 tablet_bootstrap.cc:492] T 182b3f15c06c4c07ade0a8854d3ecafd P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:15.326256 325 ts_tablet_manager.cc:1367] T 182b3f15c06c4c07ade0a8854d3ecafd P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20240502 21:26:15.326241 32765 raft_consensus.cc:504] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:15.327123 325 raft_consensus.cc:348] T 182b3f15c06c4c07ade0a8854d3ecafd P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:15.327291 32765 leader_election.cc:288] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:15.327343 325 raft_consensus.cc:374] T 182b3f15c06c4c07ade0a8854d3ecafd P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:15.327442 325 raft_consensus.cc:729] T 182b3f15c06c4c07ade0a8854d3ecafd P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:15.328083 325 consensus_queue.cc:260] T 182b3f15c06c4c07ade0a8854d3ecafd P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:15.329492 325 ts_tablet_manager.cc:1398] T 182b3f15c06c4c07ade0a8854d3ecafd P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240502 21:26:15.329784 325 tablet_bootstrap.cc:492] T b20df8896a2b423cb53c2c1a2a1b98e2 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:15.331588 325 tablet_bootstrap.cc:654] T b20df8896a2b423cb53c2c1a2a1b98e2 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:15.333809 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b20df8896a2b423cb53c2c1a2a1b98e2" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:15.333922 327 tablet_bootstrap.cc:492] T e0f96b56fb68439a97cf2aae4aa89b57 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:15.334062 327 ts_tablet_manager.cc:1367] T e0f96b56fb68439a97cf2aae4aa89b57 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.010s
I20240502 21:26:15.334793 327 raft_consensus.cc:348] T e0f96b56fb68439a97cf2aae4aa89b57 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:15.336758 327 raft_consensus.cc:374] T e0f96b56fb68439a97cf2aae4aa89b57 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:15.336848 327 raft_consensus.cc:729] T e0f96b56fb68439a97cf2aae4aa89b57 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:15.337276 327 consensus_queue.cc:260] T e0f96b56fb68439a97cf2aae4aa89b57 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:15.337949 327 ts_tablet_manager.cc:1398] T e0f96b56fb68439a97cf2aae4aa89b57 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 21:26:15.338960 325 tablet_bootstrap.cc:492] T b20df8896a2b423cb53c2c1a2a1b98e2 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:15.339151 325 ts_tablet_manager.cc:1367] T b20df8896a2b423cb53c2c1a2a1b98e2 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20240502 21:26:15.344265 325 raft_consensus.cc:348] T b20df8896a2b423cb53c2c1a2a1b98e2 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:15.344494 325 raft_consensus.cc:374] T b20df8896a2b423cb53c2c1a2a1b98e2 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:15.344576 325 raft_consensus.cc:729] T b20df8896a2b423cb53c2c1a2a1b98e2 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:15.345595 326 tablet_bootstrap.cc:492] T 182b3f15c06c4c07ade0a8854d3ecafd P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:15.345789 326 ts_tablet_manager.cc:1367] T 182b3f15c06c4c07ade0a8854d3ecafd P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.004s
W20240502 21:26:15.345758 31472 leader_election.cc:341] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:15.345821 329 raft_consensus.cc:348] T b20df8896a2b423cb53c2c1a2a1b98e2 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:15.345991 329 raft_consensus.cc:374] T b20df8896a2b423cb53c2c1a2a1b98e2 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:15.346074 329 raft_consensus.cc:729] T b20df8896a2b423cb53c2c1a2a1b98e2 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:15.346311 329 consensus_queue.cc:260] T b20df8896a2b423cb53c2c1a2a1b98e2 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:15.346706 326 raft_consensus.cc:348] T 182b3f15c06c4c07ade0a8854d3ecafd P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:15.346942 326 raft_consensus.cc:374] T 182b3f15c06c4c07ade0a8854d3ecafd P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:15.346956 329 ts_tablet_manager.cc:1398] T b20df8896a2b423cb53c2c1a2a1b98e2 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.027s user 0.001s sys 0.000s
I20240502 21:26:15.347028 326 raft_consensus.cc:729] T 182b3f15c06c4c07ade0a8854d3ecafd P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:15.347244 329 tablet_bootstrap.cc:492] T e0f96b56fb68439a97cf2aae4aa89b57 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:15.347272 326 consensus_queue.cc:260] T 182b3f15c06c4c07ade0a8854d3ecafd P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:15.347913 326 ts_tablet_manager.cc:1398] T 182b3f15c06c4c07ade0a8854d3ecafd P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:15.348506 325 consensus_queue.cc:260] T b20df8896a2b423cb53c2c1a2a1b98e2 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:15.348714 329 tablet_bootstrap.cc:654] T e0f96b56fb68439a97cf2aae4aa89b57 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:15.351778 325 ts_tablet_manager.cc:1398] T b20df8896a2b423cb53c2c1a2a1b98e2 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.013s user 0.000s sys 0.001s
I20240502 21:26:15.352228 325 tablet_bootstrap.cc:492] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:15.353672 325 tablet_bootstrap.cc:654] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:15.357406 31796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b20df8896a2b423cb53c2c1a2a1b98e2" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:15.357656 31796 raft_consensus.cc:2454] T b20df8896a2b423cb53c2c1a2a1b98e2 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:15.358078 31471 leader_election.cc:302] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f; no voters: 2d50f33557944bb480ea32199d9ee4f1
I20240502 21:26:15.358510 32765 raft_consensus.cc:2781] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:15.358616 32765 raft_consensus.cc:482] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:15.358705 32765 raft_consensus.cc:3037] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:15.361018 32765 raft_consensus.cc:504] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:15.361549 329 tablet_bootstrap.cc:492] T e0f96b56fb68439a97cf2aae4aa89b57 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:15.361668 325 tablet_bootstrap.cc:492] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:15.361786 329 ts_tablet_manager.cc:1367] T e0f96b56fb68439a97cf2aae4aa89b57 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240502 21:26:15.361820 325 ts_tablet_manager.cc:1367] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20240502 21:26:15.362198 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b20df8896a2b423cb53c2c1a2a1b98e2" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:15.362440 31926 raft_consensus.cc:3037] T b20df8896a2b423cb53c2c1a2a1b98e2 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:15.362630 325 raft_consensus.cc:348] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:15.362809 325 raft_consensus.cc:374] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:15.362900 325 raft_consensus.cc:729] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:15.362738 329 raft_consensus.cc:348] T e0f96b56fb68439a97cf2aae4aa89b57 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:15.363095 329 raft_consensus.cc:374] T e0f96b56fb68439a97cf2aae4aa89b57 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:15.363198 329 raft_consensus.cc:729] T e0f96b56fb68439a97cf2aae4aa89b57 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:15.363150 325 consensus_queue.cc:260] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:15.363495 329 consensus_queue.cc:260] T e0f96b56fb68439a97cf2aae4aa89b57 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:15.364041 329 ts_tablet_manager.cc:1398] T e0f96b56fb68439a97cf2aae4aa89b57 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:15.364933 31926 raft_consensus.cc:2454] T b20df8896a2b423cb53c2c1a2a1b98e2 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:15.365442 31472 leader_election.cc:302] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:15.365633 325 ts_tablet_manager.cc:1398] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 21:26:15.365726 317 raft_consensus.cc:2781] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:15.365870 317 raft_consensus.cc:686] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:26:15.366190 32765 leader_election.cc:288] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Requested vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:15.366187 317 consensus_queue.cc:237] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:15.366542 31796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b20df8896a2b423cb53c2c1a2a1b98e2" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:15.366698 31796 raft_consensus.cc:3037] T b20df8896a2b423cb53c2c1a2a1b98e2 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:15.369127 31796 raft_consensus.cc:2454] T b20df8896a2b423cb53c2c1a2a1b98e2 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:15.370810 31394 catalog_manager.cc:5484] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 0 to 1, leader changed from <none> to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:15.385329 32547 raft_consensus.cc:482] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:15.385484 32547 raft_consensus.cc:504] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:15.385991 32547 leader_election.cc:288] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:15.386387 31796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "182b3f15c06c4c07ade0a8854d3ecafd" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:15.386374 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "182b3f15c06c4c07ade0a8854d3ecafd" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:15.386562 31796 raft_consensus.cc:2454] T 182b3f15c06c4c07ade0a8854d3ecafd P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:26:15.386565 31535 raft_consensus.cc:2454] T 182b3f15c06c4c07ade0a8854d3ecafd P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:26:15.386904 31603 leader_election.cc:302] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:15.387153 32547 raft_consensus.cc:2781] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:15.387253 32547 raft_consensus.cc:482] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:15.387346 32547 raft_consensus.cc:3037] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:15.389050 32547 raft_consensus.cc:504] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:15.389670 32547 leader_election.cc:288] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Requested vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:15.389967 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "182b3f15c06c4c07ade0a8854d3ecafd" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:26:15.390082 31796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "182b3f15c06c4c07ade0a8854d3ecafd" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:15.390175 31535 raft_consensus.cc:3037] T 182b3f15c06c4c07ade0a8854d3ecafd P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:15.390262 31796 raft_consensus.cc:3037] T 182b3f15c06c4c07ade0a8854d3ecafd P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:15.392863 31535 raft_consensus.cc:2454] T 182b3f15c06c4c07ade0a8854d3ecafd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:15.392863 31796 raft_consensus.cc:2454] T 182b3f15c06c4c07ade0a8854d3ecafd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:15.393249 31602 leader_election.cc:302] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 08d7a0680d244b55b93d17f9ab57b84c; no voters:
I20240502 21:26:15.393481 32547 raft_consensus.cc:2781] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:15.393631 32547 raft_consensus.cc:686] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Becoming Leader. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Running, Role: LEADER
I20240502 21:26:15.393839 32547 consensus_queue.cc:237] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:15.395998 31394 catalog_manager.cc:5484] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130). New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:15.529709 32633 raft_consensus.cc:482] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:15.529883 32633 raft_consensus.cc:504] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:15.530556 32633 leader_election.cc:288] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:15.530943 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f96b56fb68439a97cf2aae4aa89b57" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:15.531153 31665 raft_consensus.cc:2454] T e0f96b56fb68439a97cf2aae4aa89b57 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:15.531106 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f96b56fb68439a97cf2aae4aa89b57" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:15.531315 31926 raft_consensus.cc:2454] T e0f96b56fb68439a97cf2aae4aa89b57 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:15.531587 31733 leader_election.cc:302] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 08d7a0680d244b55b93d17f9ab57b84c; no voters:
I20240502 21:26:15.531951 32633 raft_consensus.cc:2781] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:15.532075 32633 raft_consensus.cc:482] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:15.532169 32633 raft_consensus.cc:3037] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:15.534039 32633 raft_consensus.cc:504] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:15.534456 32633 leader_election.cc:288] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Requested vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:15.534859 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f96b56fb68439a97cf2aae4aa89b57" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:15.535003 31665 raft_consensus.cc:3037] T e0f96b56fb68439a97cf2aae4aa89b57 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:15.534976 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f96b56fb68439a97cf2aae4aa89b57" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:15.535135 31926 raft_consensus.cc:3037] T e0f96b56fb68439a97cf2aae4aa89b57 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:15.536929 31926 raft_consensus.cc:2454] T e0f96b56fb68439a97cf2aae4aa89b57 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:15.536995 31665 raft_consensus.cc:2454] T e0f96b56fb68439a97cf2aae4aa89b57 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:15.537299 31733 leader_election.cc:302] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:15.537592 32633 raft_consensus.cc:2781] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:15.537974 32633 raft_consensus.cc:686] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Becoming Leader. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Running, Role: LEADER
I20240502 21:26:15.538247 32633 consensus_queue.cc:237] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:15.540617 31394 catalog_manager.cc:5484] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: term changed from 0 to 1, leader changed from <none> to 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131). New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:15.854060 317 consensus_queue.cc:1035] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:15.859129 312 consensus_queue.cc:1035] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:15.895725 344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:15.896003 344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:15.902760 344 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 21:26:15.937019 32547 consensus_queue.cc:1035] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:15.944506 32547 consensus_queue.cc:1035] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:15.959707 32633 consensus_queue.cc:1035] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:15.966100 342 consensus_queue.cc:1035] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 189.307 ms
time per row: 0.0946533 ms
I20240502 21:26:16.236862 31392 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:16.244671 31515 tablet_service.cc:1467] Processing CreateTablet for tablet ce9c58ac50a74f8e871170936e967d64 (DEFAULT_TABLE table=rebalance_test_table_4 [id=223f9daed2194c19b553fe65892428cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:16.244966 31776 tablet_service.cc:1467] Processing CreateTablet for tablet ce9c58ac50a74f8e871170936e967d64 (DEFAULT_TABLE table=rebalance_test_table_4 [id=223f9daed2194c19b553fe65892428cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:16.245128 31515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce9c58ac50a74f8e871170936e967d64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:16.245337 31776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce9c58ac50a74f8e871170936e967d64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:16.245219 31646 tablet_service.cc:1467] Processing CreateTablet for tablet ce9c58ac50a74f8e871170936e967d64 (DEFAULT_TABLE table=rebalance_test_table_4 [id=223f9daed2194c19b553fe65892428cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:16.245668 31646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce9c58ac50a74f8e871170936e967d64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:16.245878 31777 tablet_service.cc:1467] Processing CreateTablet for tablet 0645323dfbad46ceb1e11898968e9d22 (DEFAULT_TABLE table=rebalance_test_table_4 [id=223f9daed2194c19b553fe65892428cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:16.246273 31777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0645323dfbad46ceb1e11898968e9d22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:16.252624 31906 tablet_service.cc:1467] Processing CreateTablet for tablet 69faa069cb6e4a86811ce8de61b34f90 (DEFAULT_TABLE table=rebalance_test_table_4 [id=223f9daed2194c19b553fe65892428cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:16.253072 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69faa069cb6e4a86811ce8de61b34f90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:16.253439 407 tablet_bootstrap.cc:492] T ce9c58ac50a74f8e871170936e967d64 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:16.253906 408 tablet_bootstrap.cc:492] T ce9c58ac50a74f8e871170936e967d64 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:16.254840 31645 tablet_service.cc:1467] Processing CreateTablet for tablet 69faa069cb6e4a86811ce8de61b34f90 (DEFAULT_TABLE table=rebalance_test_table_4 [id=223f9daed2194c19b553fe65892428cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:16.255016 407 tablet_bootstrap.cc:654] T ce9c58ac50a74f8e871170936e967d64 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:16.255224 31645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69faa069cb6e4a86811ce8de61b34f90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:16.256006 31516 tablet_service.cc:1467] Processing CreateTablet for tablet 0645323dfbad46ceb1e11898968e9d22 (DEFAULT_TABLE table=rebalance_test_table_4 [id=223f9daed2194c19b553fe65892428cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:16.256364 31516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0645323dfbad46ceb1e11898968e9d22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:16.257555 31907 tablet_service.cc:1467] Processing CreateTablet for tablet 0645323dfbad46ceb1e11898968e9d22 (DEFAULT_TABLE table=rebalance_test_table_4 [id=223f9daed2194c19b553fe65892428cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:16.257966 31907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0645323dfbad46ceb1e11898968e9d22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:16.259987 31515 tablet_service.cc:1467] Processing CreateTablet for tablet 69faa069cb6e4a86811ce8de61b34f90 (DEFAULT_TABLE table=rebalance_test_table_4 [id=223f9daed2194c19b553fe65892428cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:16.260372 31515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69faa069cb6e4a86811ce8de61b34f90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:16.267544 409 tablet_bootstrap.cc:492] T ce9c58ac50a74f8e871170936e967d64 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:16.269371 409 tablet_bootstrap.cc:654] T ce9c58ac50a74f8e871170936e967d64 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:16.272811 408 tablet_bootstrap.cc:654] T ce9c58ac50a74f8e871170936e967d64 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:16.279340 411 tablet_bootstrap.cc:492] T 69faa069cb6e4a86811ce8de61b34f90 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:16.279915 408 tablet_bootstrap.cc:492] T ce9c58ac50a74f8e871170936e967d64 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:16.279964 409 tablet_bootstrap.cc:492] T ce9c58ac50a74f8e871170936e967d64 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:16.280105 408 ts_tablet_manager.cc:1367] T ce9c58ac50a74f8e871170936e967d64 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.004s
I20240502 21:26:16.280109 409 ts_tablet_manager.cc:1367] T ce9c58ac50a74f8e871170936e967d64 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s
I20240502 21:26:16.280405 407 tablet_bootstrap.cc:492] T ce9c58ac50a74f8e871170936e967d64 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:16.280601 407 ts_tablet_manager.cc:1367] T ce9c58ac50a74f8e871170936e967d64 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.017s
I20240502 21:26:16.280865 409 raft_consensus.cc:348] T ce9c58ac50a74f8e871170936e967d64 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:16.280929 408 raft_consensus.cc:348] T ce9c58ac50a74f8e871170936e967d64 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:16.281093 409 raft_consensus.cc:374] T ce9c58ac50a74f8e871170936e967d64 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:16.281180 408 raft_consensus.cc:374] T ce9c58ac50a74f8e871170936e967d64 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:16.281189 409 raft_consensus.cc:729] T ce9c58ac50a74f8e871170936e967d64 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:16.281293 411 tablet_bootstrap.cc:654] T 69faa069cb6e4a86811ce8de61b34f90 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:16.281280 408 raft_consensus.cc:729] T ce9c58ac50a74f8e871170936e967d64 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:16.281415 409 consensus_queue.cc:260] T ce9c58ac50a74f8e871170936e967d64 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:16.281448 407 raft_consensus.cc:348] T ce9c58ac50a74f8e871170936e967d64 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:16.281627 407 raft_consensus.cc:374] T ce9c58ac50a74f8e871170936e967d64 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:16.281524 408 consensus_queue.cc:260] T ce9c58ac50a74f8e871170936e967d64 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:16.281715 407 raft_consensus.cc:729] T ce9c58ac50a74f8e871170936e967d64 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:16.281975 409 ts_tablet_manager.cc:1398] T ce9c58ac50a74f8e871170936e967d64 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:16.281915 407 consensus_queue.cc:260] T ce9c58ac50a74f8e871170936e967d64 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:16.282254 409 tablet_bootstrap.cc:492] T 69faa069cb6e4a86811ce8de61b34f90 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:16.282472 407 ts_tablet_manager.cc:1398] T ce9c58ac50a74f8e871170936e967d64 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:26:16.282701 407 tablet_bootstrap.cc:492] T 0645323dfbad46ceb1e11898968e9d22 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:16.284555 407 tablet_bootstrap.cc:654] T 0645323dfbad46ceb1e11898968e9d22 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:16.285919 409 tablet_bootstrap.cc:654] T 69faa069cb6e4a86811ce8de61b34f90 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:16.286679 408 ts_tablet_manager.cc:1398] T ce9c58ac50a74f8e871170936e967d64 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 21:26:16.287758 408 tablet_bootstrap.cc:492] T 0645323dfbad46ceb1e11898968e9d22 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:16.289428 408 tablet_bootstrap.cc:654] T 0645323dfbad46ceb1e11898968e9d22 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:16.292112 408 tablet_bootstrap.cc:492] T 0645323dfbad46ceb1e11898968e9d22 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:16.292398 408 ts_tablet_manager.cc:1367] T 0645323dfbad46ceb1e11898968e9d22 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240502 21:26:16.293323 408 raft_consensus.cc:348] T 0645323dfbad46ceb1e11898968e9d22 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:16.293643 408 raft_consensus.cc:374] T 0645323dfbad46ceb1e11898968e9d22 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:16.293824 408 raft_consensus.cc:729] T 0645323dfbad46ceb1e11898968e9d22 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:16.294137 408 consensus_queue.cc:260] T 0645323dfbad46ceb1e11898968e9d22 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:16.295543 408 ts_tablet_manager.cc:1398] T 0645323dfbad46ceb1e11898968e9d22 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:26:16.296722 409 tablet_bootstrap.cc:492] T 69faa069cb6e4a86811ce8de61b34f90 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:16.296891 407 tablet_bootstrap.cc:492] T 0645323dfbad46ceb1e11898968e9d22 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:16.296919 409 ts_tablet_manager.cc:1367] T 69faa069cb6e4a86811ce8de61b34f90 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20240502 21:26:16.297077 407 ts_tablet_manager.cc:1367] T 0645323dfbad46ceb1e11898968e9d22 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.006s
I20240502 21:26:16.297269 411 tablet_bootstrap.cc:492] T 69faa069cb6e4a86811ce8de61b34f90 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:16.297442 411 ts_tablet_manager.cc:1367] T 69faa069cb6e4a86811ce8de61b34f90 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.006s
I20240502 21:26:16.297755 409 raft_consensus.cc:348] T 69faa069cb6e4a86811ce8de61b34f90 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:16.297852 407 raft_consensus.cc:348] T 0645323dfbad46ceb1e11898968e9d22 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:16.298018 409 raft_consensus.cc:374] T 69faa069cb6e4a86811ce8de61b34f90 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:16.298120 407 raft_consensus.cc:374] T 0645323dfbad46ceb1e11898968e9d22 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:16.298265 409 raft_consensus.cc:729] T 69faa069cb6e4a86811ce8de61b34f90 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:16.298278 407 raft_consensus.cc:729] T 0645323dfbad46ceb1e11898968e9d22 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:16.298221 411 raft_consensus.cc:348] T 69faa069cb6e4a86811ce8de61b34f90 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:16.298496 411 raft_consensus.cc:374] T 69faa069cb6e4a86811ce8de61b34f90 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:16.298519 407 consensus_queue.cc:260] T 0645323dfbad46ceb1e11898968e9d22 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:16.298628 411 raft_consensus.cc:729] T 69faa069cb6e4a86811ce8de61b34f90 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:16.298699 409 consensus_queue.cc:260] T 69faa069cb6e4a86811ce8de61b34f90 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:16.298849 411 consensus_queue.cc:260] T 69faa069cb6e4a86811ce8de61b34f90 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:16.299099 407 ts_tablet_manager.cc:1398] T 0645323dfbad46ceb1e11898968e9d22 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:26:16.299571 411 ts_tablet_manager.cc:1398] T 69faa069cb6e4a86811ce8de61b34f90 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:16.299863 411 tablet_bootstrap.cc:492] T 0645323dfbad46ceb1e11898968e9d22 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:16.300053 409 ts_tablet_manager.cc:1398] T 69faa069cb6e4a86811ce8de61b34f90 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:26:16.300704 407 tablet_bootstrap.cc:492] T 69faa069cb6e4a86811ce8de61b34f90 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:16.301437 411 tablet_bootstrap.cc:654] T 0645323dfbad46ceb1e11898968e9d22 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:16.302338 407 tablet_bootstrap.cc:654] T 69faa069cb6e4a86811ce8de61b34f90 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:16.306361 411 tablet_bootstrap.cc:492] T 0645323dfbad46ceb1e11898968e9d22 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:16.306494 411 ts_tablet_manager.cc:1367] T 0645323dfbad46ceb1e11898968e9d22 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240502 21:26:16.306628 407 tablet_bootstrap.cc:492] T 69faa069cb6e4a86811ce8de61b34f90 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:16.306777 407 ts_tablet_manager.cc:1367] T 69faa069cb6e4a86811ce8de61b34f90 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20240502 21:26:16.306919 366 raft_consensus.cc:482] T 69faa069cb6e4a86811ce8de61b34f90 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:16.307054 366 raft_consensus.cc:504] T 69faa069cb6e4a86811ce8de61b34f90 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:16.307262 411 raft_consensus.cc:348] T 0645323dfbad46ceb1e11898968e9d22 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:16.307338 407 raft_consensus.cc:348] T 69faa069cb6e4a86811ce8de61b34f90 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:16.307431 411 raft_consensus.cc:374] T 0645323dfbad46ceb1e11898968e9d22 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:16.307462 407 raft_consensus.cc:374] T 69faa069cb6e4a86811ce8de61b34f90 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:16.307493 411 raft_consensus.cc:729] T 0645323dfbad46ceb1e11898968e9d22 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:16.307515 407 raft_consensus.cc:729] T 69faa069cb6e4a86811ce8de61b34f90 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:16.307727 366 leader_election.cc:288] T 69faa069cb6e4a86811ce8de61b34f90 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:16.307677 407 consensus_queue.cc:260] T 69faa069cb6e4a86811ce8de61b34f90 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:16.307713 411 consensus_queue.cc:260] T 0645323dfbad46ceb1e11898968e9d22 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:16.308140 407 ts_tablet_manager.cc:1398] T 69faa069cb6e4a86811ce8de61b34f90 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:26:16.308270 411 ts_tablet_manager.cc:1398] T 0645323dfbad46ceb1e11898968e9d22 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:16.308353 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69faa069cb6e4a86811ce8de61b34f90" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:16.308462 31927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69faa069cb6e4a86811ce8de61b34f90" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:16.308557 31535 raft_consensus.cc:2454] T 69faa069cb6e4a86811ce8de61b34f90 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:26:16.308665 31927 raft_consensus.cc:2454] T 69faa069cb6e4a86811ce8de61b34f90 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:26:16.308952 31603 leader_election.cc:302] T 69faa069cb6e4a86811ce8de61b34f90 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:16.309221 366 raft_consensus.cc:2781] T 69faa069cb6e4a86811ce8de61b34f90 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:16.309340 366 raft_consensus.cc:482] T 69faa069cb6e4a86811ce8de61b34f90 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:16.309419 366 raft_consensus.cc:3037] T 69faa069cb6e4a86811ce8de61b34f90 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:16.311676 366 raft_consensus.cc:504] T 69faa069cb6e4a86811ce8de61b34f90 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:16.312145 366 leader_election.cc:288] T 69faa069cb6e4a86811ce8de61b34f90 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Requested vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:16.312507 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69faa069cb6e4a86811ce8de61b34f90" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:26:16.312705 31535 raft_consensus.cc:3037] T 69faa069cb6e4a86811ce8de61b34f90 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:16.312695 31927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69faa069cb6e4a86811ce8de61b34f90" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:16.312876 31927 raft_consensus.cc:3037] T 69faa069cb6e4a86811ce8de61b34f90 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:16.315289 31535 raft_consensus.cc:2454] T 69faa069cb6e4a86811ce8de61b34f90 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:16.315366 31927 raft_consensus.cc:2454] T 69faa069cb6e4a86811ce8de61b34f90 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:16.315634 31603 leader_election.cc:302] T 69faa069cb6e4a86811ce8de61b34f90 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:16.315936 366 raft_consensus.cc:2781] T 69faa069cb6e4a86811ce8de61b34f90 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:16.316078 366 raft_consensus.cc:686] T 69faa069cb6e4a86811ce8de61b34f90 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Becoming Leader. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Running, Role: LEADER
I20240502 21:26:16.316335 366 consensus_queue.cc:237] T 69faa069cb6e4a86811ce8de61b34f90 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:16.318629 31392 catalog_manager.cc:5484] T 69faa069cb6e4a86811ce8de61b34f90 P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130). New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:16.489013 312 raft_consensus.cc:482] T ce9c58ac50a74f8e871170936e967d64 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:16.489203 312 raft_consensus.cc:504] T ce9c58ac50a74f8e871170936e967d64 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:16.489833 312 leader_election.cc:288] T ce9c58ac50a74f8e871170936e967d64 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:16.490275 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce9c58ac50a74f8e871170936e967d64" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:16.490319 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce9c58ac50a74f8e871170936e967d64" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:16.490542 31797 raft_consensus.cc:2454] T ce9c58ac50a74f8e871170936e967d64 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:16.490568 31665 raft_consensus.cc:2454] T ce9c58ac50a74f8e871170936e967d64 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:16.490936 31471 leader_election.cc:302] T ce9c58ac50a74f8e871170936e967d64 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:16.491235 312 raft_consensus.cc:2781] T ce9c58ac50a74f8e871170936e967d64 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:16.491339 312 raft_consensus.cc:482] T ce9c58ac50a74f8e871170936e967d64 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:16.491461 312 raft_consensus.cc:3037] T ce9c58ac50a74f8e871170936e967d64 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:16.493321 312 raft_consensus.cc:504] T ce9c58ac50a74f8e871170936e967d64 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:16.493846 312 leader_election.cc:288] T ce9c58ac50a74f8e871170936e967d64 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Requested vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:16.494131 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce9c58ac50a74f8e871170936e967d64" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:16.494320 31797 raft_consensus.cc:3037] T ce9c58ac50a74f8e871170936e967d64 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:16.494297 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce9c58ac50a74f8e871170936e967d64" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:16.494478 31665 raft_consensus.cc:3037] T ce9c58ac50a74f8e871170936e967d64 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:16.496068 31797 raft_consensus.cc:2454] T ce9c58ac50a74f8e871170936e967d64 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:16.496340 31665 raft_consensus.cc:2454] T ce9c58ac50a74f8e871170936e967d64 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:16.496480 31471 leader_election.cc:302] T ce9c58ac50a74f8e871170936e967d64 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:16.496733 312 raft_consensus.cc:2781] T ce9c58ac50a74f8e871170936e967d64 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:16.496881 312 raft_consensus.cc:686] T ce9c58ac50a74f8e871170936e967d64 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:26:16.497105 312 consensus_queue.cc:237] T ce9c58ac50a74f8e871170936e967d64 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:16.499565 31392 catalog_manager.cc:5484] T ce9c58ac50a74f8e871170936e967d64 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 0 to 1, leader changed from <none> to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:16.620126 312 raft_consensus.cc:482] T 0645323dfbad46ceb1e11898968e9d22 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:16.620299 312 raft_consensus.cc:504] T 0645323dfbad46ceb1e11898968e9d22 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:16.621008 312 leader_election.cc:288] T 0645323dfbad46ceb1e11898968e9d22 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:16.621358 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0645323dfbad46ceb1e11898968e9d22" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:16.621459 31927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0645323dfbad46ceb1e11898968e9d22" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:16.621551 31797 raft_consensus.cc:2454] T 0645323dfbad46ceb1e11898968e9d22 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:16.621688 31927 raft_consensus.cc:2454] T 0645323dfbad46ceb1e11898968e9d22 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:16.621912 31471 leader_election.cc:302] T 0645323dfbad46ceb1e11898968e9d22 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:16.622211 312 raft_consensus.cc:2781] T 0645323dfbad46ceb1e11898968e9d22 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:16.622300 312 raft_consensus.cc:482] T 0645323dfbad46ceb1e11898968e9d22 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:16.622359 312 raft_consensus.cc:3037] T 0645323dfbad46ceb1e11898968e9d22 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:16.623953 312 raft_consensus.cc:504] T 0645323dfbad46ceb1e11898968e9d22 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:16.624413 312 leader_election.cc:288] T 0645323dfbad46ceb1e11898968e9d22 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Requested vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:16.624812 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0645323dfbad46ceb1e11898968e9d22" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:16.624825 31927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0645323dfbad46ceb1e11898968e9d22" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:16.624966 31797 raft_consensus.cc:3037] T 0645323dfbad46ceb1e11898968e9d22 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:16.624966 31927 raft_consensus.cc:3037] T 0645323dfbad46ceb1e11898968e9d22 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:16.626605 31927 raft_consensus.cc:2454] T 0645323dfbad46ceb1e11898968e9d22 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:16.626636 31797 raft_consensus.cc:2454] T 0645323dfbad46ceb1e11898968e9d22 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:16.626914 31472 leader_election.cc:302] T 0645323dfbad46ceb1e11898968e9d22 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:16.627179 312 raft_consensus.cc:2781] T 0645323dfbad46ceb1e11898968e9d22 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:16.627298 312 raft_consensus.cc:686] T 0645323dfbad46ceb1e11898968e9d22 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:26:16.627555 312 consensus_queue.cc:237] T 0645323dfbad46ceb1e11898968e9d22 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:16.629920 31393 catalog_manager.cc:5484] T 0645323dfbad46ceb1e11898968e9d22 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 0 to 1, leader changed from <none> to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:16.810612 366 consensus_queue.cc:1035] T 69faa069cb6e4a86811ce8de61b34f90 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:16.821871 366 consensus_queue.cc:1035] T 69faa069cb6e4a86811ce8de61b34f90 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:16.927902 312 consensus_queue.cc:1035] T ce9c58ac50a74f8e871170936e967d64 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:16.933104 32765 consensus_queue.cc:1035] T ce9c58ac50a74f8e871170936e967d64 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:17.017336 32765 consensus_queue.cc:1035] T 0645323dfbad46ceb1e11898968e9d22 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:17.022740 434 consensus_queue.cc:1035] T 0645323dfbad46ceb1e11898968e9d22 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:17.039016 423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:17.039315 423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:17.046697 423 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 192.829 ms
time per row: 0.0964146 ms
I20240502 21:26:17.372365 31391 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:17.380151 31776 tablet_service.cc:1467] Processing CreateTablet for tablet ae56d18c629a4e5eb50911c89de4227c (DEFAULT_TABLE table=rebalance_test_table_5 [id=9e294586148643e080e91357bcbffb17]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:17.380486 31645 tablet_service.cc:1467] Processing CreateTablet for tablet ae56d18c629a4e5eb50911c89de4227c (DEFAULT_TABLE table=rebalance_test_table_5 [id=9e294586148643e080e91357bcbffb17]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:17.380605 31776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae56d18c629a4e5eb50911c89de4227c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:17.380852 31645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae56d18c629a4e5eb50911c89de4227c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:17.380820 31906 tablet_service.cc:1467] Processing CreateTablet for tablet ae56d18c629a4e5eb50911c89de4227c (DEFAULT_TABLE table=rebalance_test_table_5 [id=9e294586148643e080e91357bcbffb17]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:17.381242 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae56d18c629a4e5eb50911c89de4227c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:17.381160 31646 tablet_service.cc:1467] Processing CreateTablet for tablet 94ba99a5798f4d5c9442590066896f93 (DEFAULT_TABLE table=rebalance_test_table_5 [id=9e294586148643e080e91357bcbffb17]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:17.381525 31646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94ba99a5798f4d5c9442590066896f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:17.384572 31907 tablet_service.cc:1467] Processing CreateTablet for tablet 94ba99a5798f4d5c9442590066896f93 (DEFAULT_TABLE table=rebalance_test_table_5 [id=9e294586148643e080e91357bcbffb17]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:17.384891 31907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94ba99a5798f4d5c9442590066896f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:17.388494 489 tablet_bootstrap.cc:492] T ae56d18c629a4e5eb50911c89de4227c P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:17.390139 489 tablet_bootstrap.cc:654] T ae56d18c629a4e5eb50911c89de4227c P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:17.391374 31646 tablet_service.cc:1467] Processing CreateTablet for tablet a1fd14de204c40d98027ecb329ff79f3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=9e294586148643e080e91357bcbffb17]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:17.391748 31646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1fd14de204c40d98027ecb329ff79f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:17.391858 31515 tablet_service.cc:1467] Processing CreateTablet for tablet a1fd14de204c40d98027ecb329ff79f3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=9e294586148643e080e91357bcbffb17]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:17.392396 31515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1fd14de204c40d98027ecb329ff79f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:17.397601 31777 tablet_service.cc:1467] Processing CreateTablet for tablet a1fd14de204c40d98027ecb329ff79f3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=9e294586148643e080e91357bcbffb17]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:17.398038 31777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1fd14de204c40d98027ecb329ff79f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:17.400707 31776 tablet_service.cc:1467] Processing CreateTablet for tablet 94ba99a5798f4d5c9442590066896f93 (DEFAULT_TABLE table=rebalance_test_table_5 [id=9e294586148643e080e91357bcbffb17]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:17.404160 491 tablet_bootstrap.cc:492] T a1fd14de204c40d98027ecb329ff79f3 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:17.406157 491 tablet_bootstrap.cc:654] T a1fd14de204c40d98027ecb329ff79f3 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:17.406409 492 tablet_bootstrap.cc:492] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:17.407435 31776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94ba99a5798f4d5c9442590066896f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:17.408396 492 tablet_bootstrap.cc:654] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:17.415179 491 tablet_bootstrap.cc:492] T a1fd14de204c40d98027ecb329ff79f3 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:17.415822 491 ts_tablet_manager.cc:1367] T a1fd14de204c40d98027ecb329ff79f3 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20240502 21:26:17.416992 491 raft_consensus.cc:348] T a1fd14de204c40d98027ecb329ff79f3 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:17.418649 491 raft_consensus.cc:374] T a1fd14de204c40d98027ecb329ff79f3 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:17.418921 491 raft_consensus.cc:729] T a1fd14de204c40d98027ecb329ff79f3 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:17.419464 491 consensus_queue.cc:260] T a1fd14de204c40d98027ecb329ff79f3 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:17.420812 491 ts_tablet_manager.cc:1398] T a1fd14de204c40d98027ecb329ff79f3 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:26:17.421595 492 tablet_bootstrap.cc:492] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:17.421792 492 ts_tablet_manager.cc:1367] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.000s
I20240502 21:26:17.423691 488 tablet_bootstrap.cc:492] T ae56d18c629a4e5eb50911c89de4227c P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:17.422834 492 raft_consensus.cc:348] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:17.423913 492 raft_consensus.cc:374] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:17.424049 492 raft_consensus.cc:729] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:17.424472 489 tablet_bootstrap.cc:492] T ae56d18c629a4e5eb50911c89de4227c P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:17.424347 492 consensus_queue.cc:260] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:17.424713 489 ts_tablet_manager.cc:1367] T ae56d18c629a4e5eb50911c89de4227c P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.036s user 0.019s sys 0.015s
I20240502 21:26:17.425313 488 tablet_bootstrap.cc:654] T ae56d18c629a4e5eb50911c89de4227c P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:17.425453 489 raft_consensus.cc:348] T ae56d18c629a4e5eb50911c89de4227c P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:17.425664 492 ts_tablet_manager.cc:1398] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:26:17.425694 489 raft_consensus.cc:374] T ae56d18c629a4e5eb50911c89de4227c P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:17.425772 489 raft_consensus.cc:729] T ae56d18c629a4e5eb50911c89de4227c P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:17.425980 492 tablet_bootstrap.cc:492] T 94ba99a5798f4d5c9442590066896f93 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:17.426038 489 consensus_queue.cc:260] T ae56d18c629a4e5eb50911c89de4227c P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:17.427990 492 tablet_bootstrap.cc:654] T 94ba99a5798f4d5c9442590066896f93 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:17.429289 489 ts_tablet_manager.cc:1398] T ae56d18c629a4e5eb50911c89de4227c P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:17.429792 492 tablet_bootstrap.cc:492] T 94ba99a5798f4d5c9442590066896f93 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:17.429863 489 tablet_bootstrap.cc:492] T 94ba99a5798f4d5c9442590066896f93 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:17.429982 492 ts_tablet_manager.cc:1367] T 94ba99a5798f4d5c9442590066896f93 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:26:17.430773 492 raft_consensus.cc:348] T 94ba99a5798f4d5c9442590066896f93 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:17.430924 492 raft_consensus.cc:374] T 94ba99a5798f4d5c9442590066896f93 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:17.430999 492 raft_consensus.cc:729] T 94ba99a5798f4d5c9442590066896f93 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:17.431819 492 consensus_queue.cc:260] T 94ba99a5798f4d5c9442590066896f93 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:17.433142 492 ts_tablet_manager.cc:1398] T 94ba99a5798f4d5c9442590066896f93 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:26:17.433584 488 tablet_bootstrap.cc:492] T ae56d18c629a4e5eb50911c89de4227c P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:17.433734 488 ts_tablet_manager.cc:1367] T ae56d18c629a4e5eb50911c89de4227c P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20240502 21:26:17.434448 488 raft_consensus.cc:348] T ae56d18c629a4e5eb50911c89de4227c P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:17.434659 488 raft_consensus.cc:374] T ae56d18c629a4e5eb50911c89de4227c P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:17.434777 489 tablet_bootstrap.cc:654] T 94ba99a5798f4d5c9442590066896f93 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:17.434760 488 raft_consensus.cc:729] T ae56d18c629a4e5eb50911c89de4227c P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:17.435020 488 consensus_queue.cc:260] T ae56d18c629a4e5eb50911c89de4227c P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:17.435798 488 ts_tablet_manager.cc:1398] T ae56d18c629a4e5eb50911c89de4227c P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:17.437439 488 tablet_bootstrap.cc:492] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:17.439345 488 tablet_bootstrap.cc:654] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:17.441310 488 tablet_bootstrap.cc:492] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:17.441515 488 ts_tablet_manager.cc:1367] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240502 21:26:17.442301 488 raft_consensus.cc:348] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:17.442487 488 raft_consensus.cc:374] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:17.442551 488 raft_consensus.cc:729] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:17.442782 488 consensus_queue.cc:260] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:17.442914 489 tablet_bootstrap.cc:492] T 94ba99a5798f4d5c9442590066896f93 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:17.443044 489 ts_tablet_manager.cc:1367] T 94ba99a5798f4d5c9442590066896f93 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.008s
I20240502 21:26:17.443372 488 ts_tablet_manager.cc:1398] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:17.443614 488 tablet_bootstrap.cc:492] T a1fd14de204c40d98027ecb329ff79f3 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:17.443751 489 raft_consensus.cc:348] T 94ba99a5798f4d5c9442590066896f93 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:17.443939 489 raft_consensus.cc:374] T 94ba99a5798f4d5c9442590066896f93 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:17.444013 489 raft_consensus.cc:729] T 94ba99a5798f4d5c9442590066896f93 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:17.444242 489 consensus_queue.cc:260] T 94ba99a5798f4d5c9442590066896f93 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:17.444877 489 ts_tablet_manager.cc:1398] T 94ba99a5798f4d5c9442590066896f93 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:17.445026 488 tablet_bootstrap.cc:654] T a1fd14de204c40d98027ecb329ff79f3 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:17.445148 489 tablet_bootstrap.cc:492] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:17.446789 489 tablet_bootstrap.cc:654] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:17.448603 489 tablet_bootstrap.cc:492] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:17.448716 489 ts_tablet_manager.cc:1367] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240502 21:26:17.448875 488 tablet_bootstrap.cc:492] T a1fd14de204c40d98027ecb329ff79f3 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:17.449028 488 ts_tablet_manager.cc:1367] T a1fd14de204c40d98027ecb329ff79f3 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240502 21:26:17.449316 489 raft_consensus.cc:348] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:17.449461 489 raft_consensus.cc:374] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:17.449549 489 raft_consensus.cc:729] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:17.449596 488 raft_consensus.cc:348] T a1fd14de204c40d98027ecb329ff79f3 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:17.449744 488 raft_consensus.cc:374] T a1fd14de204c40d98027ecb329ff79f3 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:17.449831 488 raft_consensus.cc:729] T a1fd14de204c40d98027ecb329ff79f3 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:17.449738 489 consensus_queue.cc:260] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:17.450013 488 consensus_queue.cc:260] T a1fd14de204c40d98027ecb329ff79f3 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:17.450197 489 ts_tablet_manager.cc:1398] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:26:17.450642 488 ts_tablet_manager.cc:1398] T a1fd14de204c40d98027ecb329ff79f3 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 21:26:17.457188 32547 raft_consensus.cc:482] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:17.457295 32547 raft_consensus.cc:504] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:17.457757 32547 leader_election.cc:288] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:17.458164 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94ba99a5798f4d5c9442590066896f93" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:17.458153 31796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94ba99a5798f4d5c9442590066896f93" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:17.458326 31796 raft_consensus.cc:2454] T 94ba99a5798f4d5c9442590066896f93 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:26:17.458333 31926 raft_consensus.cc:2454] T 94ba99a5798f4d5c9442590066896f93 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:26:17.458627 31602 leader_election.cc:302] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 08d7a0680d244b55b93d17f9ab57b84c; no voters:
I20240502 21:26:17.458916 32547 raft_consensus.cc:2781] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:17.459010 32547 raft_consensus.cc:482] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:17.459100 32547 raft_consensus.cc:3037] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:17.460794 32547 raft_consensus.cc:504] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:17.461292 32547 leader_election.cc:288] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Requested vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:17.461616 31796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94ba99a5798f4d5c9442590066896f93" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:17.461710 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94ba99a5798f4d5c9442590066896f93" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:17.461786 31796 raft_consensus.cc:3037] T 94ba99a5798f4d5c9442590066896f93 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:17.461905 31926 raft_consensus.cc:3037] T 94ba99a5798f4d5c9442590066896f93 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:17.463590 31796 raft_consensus.cc:2454] T 94ba99a5798f4d5c9442590066896f93 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:17.463976 31602 leader_election.cc:302] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 08d7a0680d244b55b93d17f9ab57b84c; no voters:
I20240502 21:26:17.464126 31926 raft_consensus.cc:2454] T 94ba99a5798f4d5c9442590066896f93 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:17.464239 32547 raft_consensus.cc:2781] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:17.464383 32547 raft_consensus.cc:686] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Becoming Leader. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Running, Role: LEADER
I20240502 21:26:17.464763 32547 consensus_queue.cc:237] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:17.466869 31391 catalog_manager.cc:5484] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130). New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:17.532474 32633 raft_consensus.cc:482] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:17.532616 32633 raft_consensus.cc:504] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:17.533085 32633 leader_election.cc:288] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:17.533438 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1fd14de204c40d98027ecb329ff79f3" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:17.533490 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1fd14de204c40d98027ecb329ff79f3" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:17.533599 31665 raft_consensus.cc:2454] T a1fd14de204c40d98027ecb329ff79f3 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:17.533702 31535 raft_consensus.cc:2454] T a1fd14de204c40d98027ecb329ff79f3 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:17.533964 31733 leader_election.cc:302] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 08d7a0680d244b55b93d17f9ab57b84c; no voters:
I20240502 21:26:17.534204 32633 raft_consensus.cc:2781] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:17.534314 32633 raft_consensus.cc:482] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:17.534396 32633 raft_consensus.cc:3037] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:17.535904 32633 raft_consensus.cc:504] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:17.536345 32633 leader_election.cc:288] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Requested vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:17.536717 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1fd14de204c40d98027ecb329ff79f3" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:17.536736 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1fd14de204c40d98027ecb329ff79f3" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:26:17.536870 31665 raft_consensus.cc:3037] T a1fd14de204c40d98027ecb329ff79f3 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:17.536888 31535 raft_consensus.cc:3037] T a1fd14de204c40d98027ecb329ff79f3 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:17.538508 31665 raft_consensus.cc:2454] T a1fd14de204c40d98027ecb329ff79f3 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:17.538623 31535 raft_consensus.cc:2454] T a1fd14de204c40d98027ecb329ff79f3 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:17.538825 31733 leader_election.cc:302] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 08d7a0680d244b55b93d17f9ab57b84c; no voters:
I20240502 21:26:17.539057 32633 raft_consensus.cc:2781] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:17.539188 32633 raft_consensus.cc:686] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Becoming Leader. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Running, Role: LEADER
I20240502 21:26:17.539454 32633 consensus_queue.cc:237] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:17.541666 31391 catalog_manager.cc:5484] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: term changed from 0 to 1, leader changed from <none> to 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131). New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:17.571322 32759 raft_consensus.cc:482] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:17.571477 32759 raft_consensus.cc:504] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:17.572091 32759 leader_election.cc:288] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:17.572564 31796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae56d18c629a4e5eb50911c89de4227c" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:17.572788 31796 raft_consensus.cc:2454] T ae56d18c629a4e5eb50911c89de4227c P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:17.573161 31862 leader_election.cc:302] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:17.573513 32759 raft_consensus.cc:2781] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:17.573633 32759 raft_consensus.cc:482] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:17.573743 32759 raft_consensus.cc:3037] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:17.576088 32759 raft_consensus.cc:504] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:17.576716 32759 leader_election.cc:288] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Requested vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:17.576982 31796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae56d18c629a4e5eb50911c89de4227c" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:17.577174 31796 raft_consensus.cc:3037] T ae56d18c629a4e5eb50911c89de4227c P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:17.578440 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae56d18c629a4e5eb50911c89de4227c" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:17.578524 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae56d18c629a4e5eb50911c89de4227c" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:17.578616 31665 raft_consensus.cc:2454] T ae56d18c629a4e5eb50911c89de4227c P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:17.602222 31796 raft_consensus.cc:2454] T ae56d18c629a4e5eb50911c89de4227c P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:26:17.602593 31862 leader_election.cc:302] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 2d50f33557944bb480ea32199d9ee4f1; no voters: 05f4fac41c4f4959b9b316cb112d0ada
I20240502 21:26:17.602857 32759 raft_consensus.cc:2781] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:17.603204 32759 raft_consensus.cc:686] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Running, Role: LEADER
I20240502 21:26:17.603483 32759 consensus_queue.cc:237] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:17.605811 31391 catalog_manager.cc:5484] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132). New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:17.855808 32547 consensus_queue.cc:1035] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:17.861941 509 consensus_queue.cc:1035] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:17.933211 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:17.933544 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:17.940012 506 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240502 21:26:18.017591 483 consensus_queue.cc:1035] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:18.023141 32633 consensus_queue.cc:1035] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:18.076540 31666 raft_consensus.cc:3037] T ae56d18c629a4e5eb50911c89de4227c P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:18.087369 32759 consensus_queue.cc:1035] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:18.107056 31796 raft_consensus.cc:1261] T ae56d18c629a4e5eb50911c89de4227c P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:18.107573 32759 consensus_queue.cc:1035] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 224.463 ms
time per row: 0.112232 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 21:26:19.155174 31391 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:19.163439 31515 tablet_service.cc:1467] Processing CreateTablet for tablet b0f8779efe5549f59db5e28324df8e16 (DEFAULT_TABLE table=rebalance_test_table_6 [id=ab47909c63004e5aad56b045fb34543b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:19.163700 31645 tablet_service.cc:1467] Processing CreateTablet for tablet b0f8779efe5549f59db5e28324df8e16 (DEFAULT_TABLE table=rebalance_test_table_6 [id=ab47909c63004e5aad56b045fb34543b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:19.163898 31515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0f8779efe5549f59db5e28324df8e16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:19.164067 31645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0f8779efe5549f59db5e28324df8e16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:19.164077 31906 tablet_service.cc:1467] Processing CreateTablet for tablet b0f8779efe5549f59db5e28324df8e16 (DEFAULT_TABLE table=rebalance_test_table_6 [id=ab47909c63004e5aad56b045fb34543b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:19.164583 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0f8779efe5549f59db5e28324df8e16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:19.165132 31777 tablet_service.cc:1467] Processing CreateTablet for tablet e49ebdbe2bde42bba432662a1c7e7833 (DEFAULT_TABLE table=rebalance_test_table_6 [id=ab47909c63004e5aad56b045fb34543b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:19.165493 31777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e49ebdbe2bde42bba432662a1c7e7833. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:19.169490 31516 tablet_service.cc:1467] Processing CreateTablet for tablet e49ebdbe2bde42bba432662a1c7e7833 (DEFAULT_TABLE table=rebalance_test_table_6 [id=ab47909c63004e5aad56b045fb34543b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:19.169611 31514 tablet_service.cc:1467] Processing CreateTablet for tablet 75a35592c4b4495497967bbf256d27af (DEFAULT_TABLE table=rebalance_test_table_6 [id=ab47909c63004e5aad56b045fb34543b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:19.169873 31516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e49ebdbe2bde42bba432662a1c7e7833. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:19.169837 31646 tablet_service.cc:1467] Processing CreateTablet for tablet 75a35592c4b4495497967bbf256d27af (DEFAULT_TABLE table=rebalance_test_table_6 [id=ab47909c63004e5aad56b045fb34543b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:19.170128 31514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75a35592c4b4495497967bbf256d27af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:19.170159 31646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75a35592c4b4495497967bbf256d27af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:19.172801 31907 tablet_service.cc:1467] Processing CreateTablet for tablet e49ebdbe2bde42bba432662a1c7e7833 (DEFAULT_TABLE table=rebalance_test_table_6 [id=ab47909c63004e5aad56b045fb34543b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:19.176924 31907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e49ebdbe2bde42bba432662a1c7e7833. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:19.179440 574 tablet_bootstrap.cc:492] T b0f8779efe5549f59db5e28324df8e16 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:19.180747 576 tablet_bootstrap.cc:492] T e49ebdbe2bde42bba432662a1c7e7833 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:19.181082 31905 tablet_service.cc:1467] Processing CreateTablet for tablet 75a35592c4b4495497967bbf256d27af (DEFAULT_TABLE table=rebalance_test_table_6 [id=ab47909c63004e5aad56b045fb34543b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:19.181435 574 tablet_bootstrap.cc:654] T b0f8779efe5549f59db5e28324df8e16 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:19.181526 31905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75a35592c4b4495497967bbf256d27af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:19.182324 576 tablet_bootstrap.cc:654] T e49ebdbe2bde42bba432662a1c7e7833 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:19.186129 575 tablet_bootstrap.cc:492] T 75a35592c4b4495497967bbf256d27af P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:19.187654 575 tablet_bootstrap.cc:654] T 75a35592c4b4495497967bbf256d27af P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:19.191701 574 tablet_bootstrap.cc:492] T b0f8779efe5549f59db5e28324df8e16 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:19.191891 574 ts_tablet_manager.cc:1367] T b0f8779efe5549f59db5e28324df8e16 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20240502 21:26:19.192871 574 raft_consensus.cc:348] T b0f8779efe5549f59db5e28324df8e16 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:19.194391 574 raft_consensus.cc:374] T b0f8779efe5549f59db5e28324df8e16 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:19.194510 574 raft_consensus.cc:729] T b0f8779efe5549f59db5e28324df8e16 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:19.194783 574 consensus_queue.cc:260] T b0f8779efe5549f59db5e28324df8e16 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:19.195473 574 ts_tablet_manager.cc:1398] T b0f8779efe5549f59db5e28324df8e16 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:19.195783 574 tablet_bootstrap.cc:492] T 75a35592c4b4495497967bbf256d27af P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:19.197593 574 tablet_bootstrap.cc:654] T 75a35592c4b4495497967bbf256d27af P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:19.199679 574 tablet_bootstrap.cc:492] T 75a35592c4b4495497967bbf256d27af P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:19.199836 574 ts_tablet_manager.cc:1367] T 75a35592c4b4495497967bbf256d27af P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:26:19.200640 574 raft_consensus.cc:348] T 75a35592c4b4495497967bbf256d27af P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:19.200894 574 raft_consensus.cc:374] T 75a35592c4b4495497967bbf256d27af P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:19.200964 574 raft_consensus.cc:729] T 75a35592c4b4495497967bbf256d27af P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:19.201256 574 consensus_queue.cc:260] T 75a35592c4b4495497967bbf256d27af P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:19.202004 574 ts_tablet_manager.cc:1398] T 75a35592c4b4495497967bbf256d27af P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:19.206012 581 tablet_bootstrap.cc:492] T e49ebdbe2bde42bba432662a1c7e7833 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:19.206900 576 tablet_bootstrap.cc:492] T e49ebdbe2bde42bba432662a1c7e7833 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:19.207028 576 ts_tablet_manager.cc:1367] T e49ebdbe2bde42bba432662a1c7e7833 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.014s
I20240502 21:26:19.207384 581 tablet_bootstrap.cc:654] T e49ebdbe2bde42bba432662a1c7e7833 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:19.207564 575 tablet_bootstrap.cc:492] T 75a35592c4b4495497967bbf256d27af P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:19.207557 576 raft_consensus.cc:348] T e49ebdbe2bde42bba432662a1c7e7833 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:19.207734 575 ts_tablet_manager.cc:1367] T 75a35592c4b4495497967bbf256d27af P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.008s
I20240502 21:26:19.207753 576 raft_consensus.cc:374] T e49ebdbe2bde42bba432662a1c7e7833 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:19.207846 576 raft_consensus.cc:729] T e49ebdbe2bde42bba432662a1c7e7833 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:19.208030 576 consensus_queue.cc:260] T e49ebdbe2bde42bba432662a1c7e7833 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:19.208460 575 raft_consensus.cc:348] T 75a35592c4b4495497967bbf256d27af P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:19.208628 576 ts_tablet_manager.cc:1398] T e49ebdbe2bde42bba432662a1c7e7833 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:19.208662 575 raft_consensus.cc:374] T 75a35592c4b4495497967bbf256d27af P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:19.208747 575 raft_consensus.cc:729] T 75a35592c4b4495497967bbf256d27af P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:19.208992 575 consensus_queue.cc:260] T 75a35592c4b4495497967bbf256d27af P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:19.209795 581 tablet_bootstrap.cc:492] T e49ebdbe2bde42bba432662a1c7e7833 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:19.209852 575 ts_tablet_manager.cc:1398] T 75a35592c4b4495497967bbf256d27af P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:19.209923 581 ts_tablet_manager.cc:1367] T e49ebdbe2bde42bba432662a1c7e7833 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:26:19.210083 575 tablet_bootstrap.cc:492] T b0f8779efe5549f59db5e28324df8e16 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:19.211668 575 tablet_bootstrap.cc:654] T b0f8779efe5549f59db5e28324df8e16 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:19.211652 581 raft_consensus.cc:348] T e49ebdbe2bde42bba432662a1c7e7833 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:19.211863 581 raft_consensus.cc:374] T e49ebdbe2bde42bba432662a1c7e7833 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:19.211969 581 raft_consensus.cc:729] T e49ebdbe2bde42bba432662a1c7e7833 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:19.212204 581 consensus_queue.cc:260] T e49ebdbe2bde42bba432662a1c7e7833 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:19.213385 575 tablet_bootstrap.cc:492] T b0f8779efe5549f59db5e28324df8e16 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:19.213531 575 ts_tablet_manager.cc:1367] T b0f8779efe5549f59db5e28324df8e16 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 21:26:19.214196 581 ts_tablet_manager.cc:1398] T e49ebdbe2bde42bba432662a1c7e7833 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:26:19.214112 575 raft_consensus.cc:348] T b0f8779efe5549f59db5e28324df8e16 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:19.214289 575 raft_consensus.cc:374] T b0f8779efe5549f59db5e28324df8e16 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:19.214359 575 raft_consensus.cc:729] T b0f8779efe5549f59db5e28324df8e16 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:19.214440 581 tablet_bootstrap.cc:492] T b0f8779efe5549f59db5e28324df8e16 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:19.214579 575 consensus_queue.cc:260] T b0f8779efe5549f59db5e28324df8e16 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:19.215077 575 ts_tablet_manager.cc:1398] T b0f8779efe5549f59db5e28324df8e16 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:26:19.215330 575 tablet_bootstrap.cc:492] T e49ebdbe2bde42bba432662a1c7e7833 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:19.216449 581 tablet_bootstrap.cc:654] T b0f8779efe5549f59db5e28324df8e16 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:19.216629 575 tablet_bootstrap.cc:654] T e49ebdbe2bde42bba432662a1c7e7833 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:19.218875 575 tablet_bootstrap.cc:492] T e49ebdbe2bde42bba432662a1c7e7833 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:19.219023 575 ts_tablet_manager.cc:1367] T e49ebdbe2bde42bba432662a1c7e7833 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:26:19.219974 575 raft_consensus.cc:348] T e49ebdbe2bde42bba432662a1c7e7833 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:19.220191 575 raft_consensus.cc:374] T e49ebdbe2bde42bba432662a1c7e7833 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:19.220288 575 raft_consensus.cc:729] T e49ebdbe2bde42bba432662a1c7e7833 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:19.220542 575 consensus_queue.cc:260] T e49ebdbe2bde42bba432662a1c7e7833 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:19.221132 575 ts_tablet_manager.cc:1398] T e49ebdbe2bde42bba432662a1c7e7833 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:19.222108 581 tablet_bootstrap.cc:492] T b0f8779efe5549f59db5e28324df8e16 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:19.222234 581 ts_tablet_manager.cc:1367] T b0f8779efe5549f59db5e28324df8e16 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20240502 21:26:19.222872 581 raft_consensus.cc:348] T b0f8779efe5549f59db5e28324df8e16 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:19.223083 581 raft_consensus.cc:374] T b0f8779efe5549f59db5e28324df8e16 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:19.223165 581 raft_consensus.cc:729] T b0f8779efe5549f59db5e28324df8e16 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:19.223372 581 consensus_queue.cc:260] T b0f8779efe5549f59db5e28324df8e16 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:19.223754 581 ts_tablet_manager.cc:1398] T b0f8779efe5549f59db5e28324df8e16 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:26:19.223989 581 tablet_bootstrap.cc:492] T 75a35592c4b4495497967bbf256d27af P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:19.225061 581 tablet_bootstrap.cc:654] T 75a35592c4b4495497967bbf256d27af P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:19.226317 581 tablet_bootstrap.cc:492] T 75a35592c4b4495497967bbf256d27af P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:19.226423 581 ts_tablet_manager.cc:1367] T 75a35592c4b4495497967bbf256d27af P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:19.226883 581 raft_consensus.cc:348] T 75a35592c4b4495497967bbf256d27af P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:19.227023 581 raft_consensus.cc:374] T 75a35592c4b4495497967bbf256d27af P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:19.227104 581 raft_consensus.cc:729] T 75a35592c4b4495497967bbf256d27af P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:19.227278 581 consensus_queue.cc:260] T 75a35592c4b4495497967bbf256d27af P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:19.227635 581 ts_tablet_manager.cc:1398] T 75a35592c4b4495497967bbf256d27af P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:26:19.238356 509 raft_consensus.cc:482] T 75a35592c4b4495497967bbf256d27af P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:19.238499 509 raft_consensus.cc:504] T 75a35592c4b4495497967bbf256d27af P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:19.238936 509 leader_election.cc:288] T 75a35592c4b4495497967bbf256d27af P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:19.239271 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75a35592c4b4495497967bbf256d27af" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:19.239329 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75a35592c4b4495497967bbf256d27af" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:19.239418 31926 raft_consensus.cc:2454] T 75a35592c4b4495497967bbf256d27af P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:26:19.239521 31535 raft_consensus.cc:2454] T 75a35592c4b4495497967bbf256d27af P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:26:19.239742 31603 leader_election.cc:302] T 75a35592c4b4495497967bbf256d27af P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:19.239959 509 raft_consensus.cc:2781] T 75a35592c4b4495497967bbf256d27af P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:19.240031 509 raft_consensus.cc:482] T 75a35592c4b4495497967bbf256d27af P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:19.240125 509 raft_consensus.cc:3037] T 75a35592c4b4495497967bbf256d27af P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:19.241730 509 raft_consensus.cc:504] T 75a35592c4b4495497967bbf256d27af P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:19.242278 509 leader_election.cc:288] T 75a35592c4b4495497967bbf256d27af P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Requested vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:19.242583 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75a35592c4b4495497967bbf256d27af" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:19.242712 31926 raft_consensus.cc:3037] T 75a35592c4b4495497967bbf256d27af P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:19.242682 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75a35592c4b4495497967bbf256d27af" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:26:19.242851 31535 raft_consensus.cc:3037] T 75a35592c4b4495497967bbf256d27af P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:19.244298 31926 raft_consensus.cc:2454] T 75a35592c4b4495497967bbf256d27af P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:19.244391 31535 raft_consensus.cc:2454] T 75a35592c4b4495497967bbf256d27af P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:19.244658 31603 leader_election.cc:302] T 75a35592c4b4495497967bbf256d27af P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:19.244858 509 raft_consensus.cc:2781] T 75a35592c4b4495497967bbf256d27af P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:19.245160 509 raft_consensus.cc:686] T 75a35592c4b4495497967bbf256d27af P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Becoming Leader. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Running, Role: LEADER
I20240502 21:26:19.245429 509 consensus_queue.cc:237] T 75a35592c4b4495497967bbf256d27af P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:19.247310 31391 catalog_manager.cc:5484] T 75a35592c4b4495497967bbf256d27af P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130). New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:19.359632 509 raft_consensus.cc:482] T b0f8779efe5549f59db5e28324df8e16 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:19.359757 509 raft_consensus.cc:504] T b0f8779efe5549f59db5e28324df8e16 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:19.360276 509 leader_election.cc:288] T b0f8779efe5549f59db5e28324df8e16 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:19.360664 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8779efe5549f59db5e28324df8e16" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:19.360699 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8779efe5549f59db5e28324df8e16" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:19.360886 31926 raft_consensus.cc:2454] T b0f8779efe5549f59db5e28324df8e16 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:26:19.360886 31535 raft_consensus.cc:2454] T b0f8779efe5549f59db5e28324df8e16 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:26:19.361207 31603 leader_election.cc:302] T b0f8779efe5549f59db5e28324df8e16 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:19.361456 509 raft_consensus.cc:2781] T b0f8779efe5549f59db5e28324df8e16 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:19.361594 509 raft_consensus.cc:482] T b0f8779efe5549f59db5e28324df8e16 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:19.361673 509 raft_consensus.cc:3037] T b0f8779efe5549f59db5e28324df8e16 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:19.363330 509 raft_consensus.cc:504] T b0f8779efe5549f59db5e28324df8e16 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:19.363777 509 leader_election.cc:288] T b0f8779efe5549f59db5e28324df8e16 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Requested vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:19.364084 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8779efe5549f59db5e28324df8e16" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:26:19.364151 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f8779efe5549f59db5e28324df8e16" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:19.364235 31535 raft_consensus.cc:3037] T b0f8779efe5549f59db5e28324df8e16 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:19.364353 31926 raft_consensus.cc:3037] T b0f8779efe5549f59db5e28324df8e16 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:19.365947 31535 raft_consensus.cc:2454] T b0f8779efe5549f59db5e28324df8e16 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:19.366055 31926 raft_consensus.cc:2454] T b0f8779efe5549f59db5e28324df8e16 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:19.366291 31603 leader_election.cc:302] T b0f8779efe5549f59db5e28324df8e16 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:19.366554 509 raft_consensus.cc:2781] T b0f8779efe5549f59db5e28324df8e16 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:19.366714 509 raft_consensus.cc:686] T b0f8779efe5549f59db5e28324df8e16 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Becoming Leader. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Running, Role: LEADER
I20240502 21:26:19.367002 509 consensus_queue.cc:237] T b0f8779efe5549f59db5e28324df8e16 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:19.369524 31391 catalog_manager.cc:5484] T b0f8779efe5549f59db5e28324df8e16 P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130). New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:19.585033 559 raft_consensus.cc:482] T e49ebdbe2bde42bba432662a1c7e7833 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:19.585228 559 raft_consensus.cc:504] T e49ebdbe2bde42bba432662a1c7e7833 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:19.585798 559 leader_election.cc:288] T e49ebdbe2bde42bba432662a1c7e7833 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:19.586153 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e49ebdbe2bde42bba432662a1c7e7833" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:19.586234 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e49ebdbe2bde42bba432662a1c7e7833" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:19.586380 31797 raft_consensus.cc:2454] T e49ebdbe2bde42bba432662a1c7e7833 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:19.586426 31535 raft_consensus.cc:2454] T e49ebdbe2bde42bba432662a1c7e7833 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:19.586783 31862 leader_election.cc:302] T e49ebdbe2bde42bba432662a1c7e7833 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 2d50f33557944bb480ea32199d9ee4f1, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:19.587020 559 raft_consensus.cc:2781] T e49ebdbe2bde42bba432662a1c7e7833 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:19.587141 559 raft_consensus.cc:482] T e49ebdbe2bde42bba432662a1c7e7833 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:19.587225 559 raft_consensus.cc:3037] T e49ebdbe2bde42bba432662a1c7e7833 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:19.589033 559 raft_consensus.cc:504] T e49ebdbe2bde42bba432662a1c7e7833 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:19.589543 559 leader_election.cc:288] T e49ebdbe2bde42bba432662a1c7e7833 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Requested vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:19.589956 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e49ebdbe2bde42bba432662a1c7e7833" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:26:19.589985 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e49ebdbe2bde42bba432662a1c7e7833" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:19.590143 31535 raft_consensus.cc:3037] T e49ebdbe2bde42bba432662a1c7e7833 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:19.590178 31797 raft_consensus.cc:3037] T e49ebdbe2bde42bba432662a1c7e7833 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:19.593376 31797 raft_consensus.cc:2454] T e49ebdbe2bde42bba432662a1c7e7833 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:26:19.593391 31535 raft_consensus.cc:2454] T e49ebdbe2bde42bba432662a1c7e7833 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:26:19.593766 31863 leader_election.cc:302] T e49ebdbe2bde42bba432662a1c7e7833 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 2d50f33557944bb480ea32199d9ee4f1, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:19.594056 559 raft_consensus.cc:2781] T e49ebdbe2bde42bba432662a1c7e7833 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:19.594341 559 raft_consensus.cc:686] T e49ebdbe2bde42bba432662a1c7e7833 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Running, Role: LEADER
I20240502 21:26:19.594569 559 consensus_queue.cc:237] T e49ebdbe2bde42bba432662a1c7e7833 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:19.596923 31391 catalog_manager.cc:5484] T e49ebdbe2bde42bba432662a1c7e7833 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132). New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:19.645668 509 consensus_queue.cc:1035] T 75a35592c4b4495497967bbf256d27af P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:19.650804 587 consensus_queue.cc:1035] T 75a35592c4b4495497967bbf256d27af P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:19.936172 592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:19.936444 592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:19.943699 592 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:26:19.959704 587 consensus_queue.cc:1035] T b0f8779efe5549f59db5e28324df8e16 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:19.966066 587 consensus_queue.cc:1035] T b0f8779efe5549f59db5e28324df8e16 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_6'
I20240502 21:26:19.981701 559 consensus_queue.cc:1035] T e49ebdbe2bde42bba432662a1c7e7833 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:19.989110 590 consensus_queue.cc:1035] T e49ebdbe2bde42bba432662a1c7e7833 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 198.813 ms
time per row: 0.0994065 ms
I20240502 21:26:20.304865 31389 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:20.312690 31777 tablet_service.cc:1467] Processing CreateTablet for tablet 44277dc39b8a4729b704986edbfb9b84 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0af22f3605164ff58dc3914737c2de4f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:20.313014 31646 tablet_service.cc:1467] Processing CreateTablet for tablet 44277dc39b8a4729b704986edbfb9b84 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0af22f3605164ff58dc3914737c2de4f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:20.313134 31777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44277dc39b8a4729b704986edbfb9b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:20.313364 31646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44277dc39b8a4729b704986edbfb9b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:20.314288 31645 tablet_service.cc:1467] Processing CreateTablet for tablet f9bbc56ba2f841b9bb36a864e1a1f808 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0af22f3605164ff58dc3914737c2de4f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:20.314478 31776 tablet_service.cc:1467] Processing CreateTablet for tablet f9bbc56ba2f841b9bb36a864e1a1f808 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0af22f3605164ff58dc3914737c2de4f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:20.314608 31645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9bbc56ba2f841b9bb36a864e1a1f808. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:20.314828 31776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9bbc56ba2f841b9bb36a864e1a1f808. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:20.317162 31775 tablet_service.cc:1467] Processing CreateTablet for tablet 53b7eec870c44999b48f0bf744b2ffcd (DEFAULT_TABLE table=rebalance_test_table_7 [id=0af22f3605164ff58dc3914737c2de4f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:20.317492 31775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b7eec870c44999b48f0bf744b2ffcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:20.319391 31907 tablet_service.cc:1467] Processing CreateTablet for tablet f9bbc56ba2f841b9bb36a864e1a1f808 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0af22f3605164ff58dc3914737c2de4f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:20.319643 31905 tablet_service.cc:1467] Processing CreateTablet for tablet 53b7eec870c44999b48f0bf744b2ffcd (DEFAULT_TABLE table=rebalance_test_table_7 [id=0af22f3605164ff58dc3914737c2de4f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:20.319789 31907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9bbc56ba2f841b9bb36a864e1a1f808. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:20.319955 31905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b7eec870c44999b48f0bf744b2ffcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:20.320880 31514 tablet_service.cc:1467] Processing CreateTablet for tablet 44277dc39b8a4729b704986edbfb9b84 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0af22f3605164ff58dc3914737c2de4f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:20.321416 31514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44277dc39b8a4729b704986edbfb9b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:20.324836 31516 tablet_service.cc:1467] Processing CreateTablet for tablet 53b7eec870c44999b48f0bf744b2ffcd (DEFAULT_TABLE table=rebalance_test_table_7 [id=0af22f3605164ff58dc3914737c2de4f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:20.325227 31516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b7eec870c44999b48f0bf744b2ffcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:20.331293 653 tablet_bootstrap.cc:492] T 44277dc39b8a4729b704986edbfb9b84 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:20.334484 654 tablet_bootstrap.cc:492] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:20.335768 654 tablet_bootstrap.cc:654] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:20.333542 653 tablet_bootstrap.cc:654] T 44277dc39b8a4729b704986edbfb9b84 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:20.337633 654 tablet_bootstrap.cc:492] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:20.337785 654 ts_tablet_manager.cc:1367] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:26:20.338315 654 raft_consensus.cc:348] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:20.338464 654 raft_consensus.cc:374] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:20.338552 654 raft_consensus.cc:729] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:20.338748 654 consensus_queue.cc:260] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:20.339241 654 ts_tablet_manager.cc:1398] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 21:26:20.339632 654 tablet_bootstrap.cc:492] T 53b7eec870c44999b48f0bf744b2ffcd P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:20.341065 654 tablet_bootstrap.cc:654] T 53b7eec870c44999b48f0bf744b2ffcd P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:20.349200 654 tablet_bootstrap.cc:492] T 53b7eec870c44999b48f0bf744b2ffcd P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:20.349349 654 ts_tablet_manager.cc:1367] T 53b7eec870c44999b48f0bf744b2ffcd P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20240502 21:26:20.349915 654 raft_consensus.cc:348] T 53b7eec870c44999b48f0bf744b2ffcd P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:20.350064 654 raft_consensus.cc:374] T 53b7eec870c44999b48f0bf744b2ffcd P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:20.350147 654 raft_consensus.cc:729] T 53b7eec870c44999b48f0bf744b2ffcd P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:20.350345 654 consensus_queue.cc:260] T 53b7eec870c44999b48f0bf744b2ffcd P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:20.350744 654 ts_tablet_manager.cc:1398] T 53b7eec870c44999b48f0bf744b2ffcd P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:26:20.358256 657 tablet_bootstrap.cc:492] T 53b7eec870c44999b48f0bf744b2ffcd P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:20.360342 657 tablet_bootstrap.cc:654] T 53b7eec870c44999b48f0bf744b2ffcd P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:20.362003 657 tablet_bootstrap.cc:492] T 53b7eec870c44999b48f0bf744b2ffcd P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:20.362133 657 ts_tablet_manager.cc:1367] T 53b7eec870c44999b48f0bf744b2ffcd P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:26:20.362700 657 raft_consensus.cc:348] T 53b7eec870c44999b48f0bf744b2ffcd P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:20.362839 657 raft_consensus.cc:374] T 53b7eec870c44999b48f0bf744b2ffcd P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:20.362949 657 raft_consensus.cc:729] T 53b7eec870c44999b48f0bf744b2ffcd P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:20.363174 657 consensus_queue.cc:260] T 53b7eec870c44999b48f0bf744b2ffcd P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:20.364606 657 ts_tablet_manager.cc:1398] T 53b7eec870c44999b48f0bf744b2ffcd P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:20.364884 657 tablet_bootstrap.cc:492] T 44277dc39b8a4729b704986edbfb9b84 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:20.365343 653 tablet_bootstrap.cc:492] T 44277dc39b8a4729b704986edbfb9b84 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:20.365602 653 ts_tablet_manager.cc:1367] T 44277dc39b8a4729b704986edbfb9b84 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.002s
I20240502 21:26:20.366503 653 raft_consensus.cc:348] T 44277dc39b8a4729b704986edbfb9b84 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:20.366786 657 tablet_bootstrap.cc:654] T 44277dc39b8a4729b704986edbfb9b84 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:20.366801 653 raft_consensus.cc:374] T 44277dc39b8a4729b704986edbfb9b84 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:20.366982 653 raft_consensus.cc:729] T 44277dc39b8a4729b704986edbfb9b84 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:20.367280 653 consensus_queue.cc:260] T 44277dc39b8a4729b704986edbfb9b84 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:20.369081 653 ts_tablet_manager.cc:1398] T 44277dc39b8a4729b704986edbfb9b84 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:26:20.369311 653 tablet_bootstrap.cc:492] T f9bbc56ba2f841b9bb36a864e1a1f808 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:20.370079 657 tablet_bootstrap.cc:492] T 44277dc39b8a4729b704986edbfb9b84 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:20.370205 657 ts_tablet_manager.cc:1367] T 44277dc39b8a4729b704986edbfb9b84 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240502 21:26:20.371181 653 tablet_bootstrap.cc:654] T f9bbc56ba2f841b9bb36a864e1a1f808 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:20.372686 657 raft_consensus.cc:348] T 44277dc39b8a4729b704986edbfb9b84 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:20.372859 657 raft_consensus.cc:374] T 44277dc39b8a4729b704986edbfb9b84 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:20.372927 657 raft_consensus.cc:729] T 44277dc39b8a4729b704986edbfb9b84 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:20.373129 657 consensus_queue.cc:260] T 44277dc39b8a4729b704986edbfb9b84 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:20.374017 657 ts_tablet_manager.cc:1398] T 44277dc39b8a4729b704986edbfb9b84 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 21:26:20.375445 652 tablet_bootstrap.cc:492] T 44277dc39b8a4729b704986edbfb9b84 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:20.376917 652 tablet_bootstrap.cc:654] T 44277dc39b8a4729b704986edbfb9b84 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:20.378437 653 tablet_bootstrap.cc:492] T f9bbc56ba2f841b9bb36a864e1a1f808 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:20.378610 652 tablet_bootstrap.cc:492] T 44277dc39b8a4729b704986edbfb9b84 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:20.378605 653 ts_tablet_manager.cc:1367] T f9bbc56ba2f841b9bb36a864e1a1f808 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240502 21:26:20.378738 652 ts_tablet_manager.cc:1367] T 44277dc39b8a4729b704986edbfb9b84 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:20.379204 653 raft_consensus.cc:348] T f9bbc56ba2f841b9bb36a864e1a1f808 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:20.379385 653 raft_consensus.cc:374] T f9bbc56ba2f841b9bb36a864e1a1f808 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:20.379351 652 raft_consensus.cc:348] T 44277dc39b8a4729b704986edbfb9b84 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:20.379470 653 raft_consensus.cc:729] T f9bbc56ba2f841b9bb36a864e1a1f808 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:20.379535 652 raft_consensus.cc:374] T 44277dc39b8a4729b704986edbfb9b84 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:20.379653 652 raft_consensus.cc:729] T 44277dc39b8a4729b704986edbfb9b84 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:20.379675 653 consensus_queue.cc:260] T f9bbc56ba2f841b9bb36a864e1a1f808 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:20.379874 652 consensus_queue.cc:260] T 44277dc39b8a4729b704986edbfb9b84 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:20.380654 652 ts_tablet_manager.cc:1398] T 44277dc39b8a4729b704986edbfb9b84 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:20.381062 653 ts_tablet_manager.cc:1398] T f9bbc56ba2f841b9bb36a864e1a1f808 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:20.381099 652 tablet_bootstrap.cc:492] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:20.382818 652 tablet_bootstrap.cc:654] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:20.386464 652 tablet_bootstrap.cc:492] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:20.386587 652 ts_tablet_manager.cc:1367] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20240502 21:26:20.387060 652 raft_consensus.cc:348] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:20.387212 652 raft_consensus.cc:374] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:20.387269 652 raft_consensus.cc:729] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:20.387436 652 consensus_queue.cc:260] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:20.387765 652 ts_tablet_manager.cc:1398] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:26:20.387993 652 tablet_bootstrap.cc:492] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:20.389321 652 tablet_bootstrap.cc:654] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:20.390805 652 tablet_bootstrap.cc:492] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:20.391144 652 ts_tablet_manager.cc:1367] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20240502 21:26:20.391930 652 raft_consensus.cc:348] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:20.392078 652 raft_consensus.cc:374] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:20.392143 652 raft_consensus.cc:729] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:20.392303 652 consensus_queue.cc:260] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:20.392689 652 ts_tablet_manager.cc:1398] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 21:26:20.428457 598 raft_consensus.cc:482] T 44277dc39b8a4729b704986edbfb9b84 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:20.428582 598 raft_consensus.cc:504] T 44277dc39b8a4729b704986edbfb9b84 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:20.429100 598 leader_election.cc:288] T 44277dc39b8a4729b704986edbfb9b84 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:20.429471 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44277dc39b8a4729b704986edbfb9b84" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:20.429488 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44277dc39b8a4729b704986edbfb9b84" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:20.429646 31536 raft_consensus.cc:2454] T 44277dc39b8a4729b704986edbfb9b84 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:20.429643 31666 raft_consensus.cc:2454] T 44277dc39b8a4729b704986edbfb9b84 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:20.429989 31733 leader_election.cc:302] T 44277dc39b8a4729b704986edbfb9b84 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:20.430210 598 raft_consensus.cc:2781] T 44277dc39b8a4729b704986edbfb9b84 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:20.430318 598 raft_consensus.cc:482] T 44277dc39b8a4729b704986edbfb9b84 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:20.430418 598 raft_consensus.cc:3037] T 44277dc39b8a4729b704986edbfb9b84 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:20.432013 598 raft_consensus.cc:504] T 44277dc39b8a4729b704986edbfb9b84 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:20.432518 598 leader_election.cc:288] T 44277dc39b8a4729b704986edbfb9b84 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Requested vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:20.432829 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44277dc39b8a4729b704986edbfb9b84" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:20.432992 31666 raft_consensus.cc:3037] T 44277dc39b8a4729b704986edbfb9b84 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:20.432963 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44277dc39b8a4729b704986edbfb9b84" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:26:20.433118 31536 raft_consensus.cc:3037] T 44277dc39b8a4729b704986edbfb9b84 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:20.434583 31666 raft_consensus.cc:2454] T 44277dc39b8a4729b704986edbfb9b84 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:20.434707 31536 raft_consensus.cc:2454] T 44277dc39b8a4729b704986edbfb9b84 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:20.434875 31733 leader_election.cc:302] T 44277dc39b8a4729b704986edbfb9b84 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 08d7a0680d244b55b93d17f9ab57b84c; no voters:
I20240502 21:26:20.435140 598 raft_consensus.cc:2781] T 44277dc39b8a4729b704986edbfb9b84 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:20.435267 598 raft_consensus.cc:686] T 44277dc39b8a4729b704986edbfb9b84 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Becoming Leader. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Running, Role: LEADER
I20240502 21:26:20.435506 598 consensus_queue.cc:237] T 44277dc39b8a4729b704986edbfb9b84 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:20.437530 31389 catalog_manager.cc:5484] T 44277dc39b8a4729b704986edbfb9b84 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: term changed from 0 to 1, leader changed from <none> to 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131). New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:20.528573 598 raft_consensus.cc:482] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:20.528731 598 raft_consensus.cc:504] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:20.529345 598 leader_election.cc:288] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:20.529827 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9bbc56ba2f841b9bb36a864e1a1f808" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:20.529857 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9bbc56ba2f841b9bb36a864e1a1f808" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:20.530052 31666 raft_consensus.cc:2454] T f9bbc56ba2f841b9bb36a864e1a1f808 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:20.530071 31926 raft_consensus.cc:2454] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:20.530442 31733 leader_election.cc:302] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 08d7a0680d244b55b93d17f9ab57b84c; no voters:
I20240502 21:26:20.530685 598 raft_consensus.cc:2781] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:20.530774 598 raft_consensus.cc:482] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:20.530872 598 raft_consensus.cc:3037] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:20.533106 598 raft_consensus.cc:504] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:20.533761 598 leader_election.cc:288] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Requested vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:20.534046 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9bbc56ba2f841b9bb36a864e1a1f808" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:20.534107 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9bbc56ba2f841b9bb36a864e1a1f808" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:20.534219 31666 raft_consensus.cc:3037] T f9bbc56ba2f841b9bb36a864e1a1f808 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:20.534301 31926 raft_consensus.cc:3037] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:20.536429 31666 raft_consensus.cc:2454] T f9bbc56ba2f841b9bb36a864e1a1f808 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:20.536597 31926 raft_consensus.cc:2454] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:20.536816 31733 leader_election.cc:302] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 08d7a0680d244b55b93d17f9ab57b84c; no voters:
I20240502 21:26:20.537057 598 raft_consensus.cc:2781] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:20.537214 598 raft_consensus.cc:686] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Becoming Leader. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Running, Role: LEADER
I20240502 21:26:20.537452 598 consensus_queue.cc:237] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:20.540020 31389 catalog_manager.cc:5484] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: term changed from 0 to 1, leader changed from <none> to 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131). New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:20.615429 598 raft_consensus.cc:482] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:20.615582 598 raft_consensus.cc:504] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:20.616104 598 leader_election.cc:288] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:20.616393 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b7eec870c44999b48f0bf744b2ffcd" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:20.616511 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b7eec870c44999b48f0bf744b2ffcd" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:20.616690 31926 raft_consensus.cc:2454] T 53b7eec870c44999b48f0bf744b2ffcd P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:20.616748 31536 raft_consensus.cc:2454] T 53b7eec870c44999b48f0bf744b2ffcd P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:20.617066 31733 leader_election.cc:302] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:20.617306 598 raft_consensus.cc:2781] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:20.617435 598 raft_consensus.cc:482] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:20.617516 598 raft_consensus.cc:3037] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:20.619212 598 raft_consensus.cc:504] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:20.619683 598 leader_election.cc:288] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Requested vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:20.619962 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b7eec870c44999b48f0bf744b2ffcd" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:20.620121 31926 raft_consensus.cc:3037] T 53b7eec870c44999b48f0bf744b2ffcd P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:20.620134 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b7eec870c44999b48f0bf744b2ffcd" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:26:20.620329 31536 raft_consensus.cc:3037] T 53b7eec870c44999b48f0bf744b2ffcd P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:20.621918 31926 raft_consensus.cc:2454] T 53b7eec870c44999b48f0bf744b2ffcd P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:20.622155 31536 raft_consensus.cc:2454] T 53b7eec870c44999b48f0bf744b2ffcd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:20.622267 31733 leader_election.cc:302] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:20.622602 598 raft_consensus.cc:2781] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:20.623011 598 raft_consensus.cc:686] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Becoming Leader. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Running, Role: LEADER
I20240502 21:26:20.623281 598 consensus_queue.cc:237] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:20.625808 31389 catalog_manager.cc:5484] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: term changed from 0 to 1, leader changed from <none> to 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131). New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:20.901387 598 consensus_queue.cc:1035] T 44277dc39b8a4729b704986edbfb9b84 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:20.912914 668 consensus_queue.cc:1035] T 44277dc39b8a4729b704986edbfb9b84 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:20.945819 670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:20.946064 670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:20.953557 670 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240502 21:26:21.066004 31926 raft_consensus.cc:1261] T 53b7eec870c44999b48f0bf744b2ffcd P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:21.066272 31536 raft_consensus.cc:1261] T 53b7eec870c44999b48f0bf744b2ffcd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:21.066820 665 consensus_queue.cc:1035] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:21.067301 665 consensus_queue.cc:1035] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:21.072387 31666 raft_consensus.cc:1261] T f9bbc56ba2f841b9bb36a864e1a1f808 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:21.074268 598 consensus_queue.cc:1035] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:21.075449 31927 raft_consensus.cc:1261] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:21.076516 668 consensus_queue.cc:1035] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 164.282 ms
time per row: 0.082141 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 21:26:22.140970 31393 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:22.149425 31777 tablet_service.cc:1467] Processing CreateTablet for tablet 656e698c34394c38a723230d788b702a (DEFAULT_TABLE table=rebalance_test_table_8 [id=f3134ee4e058445b955a09231a5e159f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:22.149878 31777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 656e698c34394c38a723230d788b702a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:22.149919 31514 tablet_service.cc:1467] Processing CreateTablet for tablet 656e698c34394c38a723230d788b702a (DEFAULT_TABLE table=rebalance_test_table_8 [id=f3134ee4e058445b955a09231a5e159f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:22.150346 31514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 656e698c34394c38a723230d788b702a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:22.150599 31905 tablet_service.cc:1467] Processing CreateTablet for tablet 98e66cb7daff4977927911bbfad93b62 (DEFAULT_TABLE table=rebalance_test_table_8 [id=f3134ee4e058445b955a09231a5e159f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:22.150842 31907 tablet_service.cc:1467] Processing CreateTablet for tablet 930b5fcce62a4040a2e2b0f0f9462477 (DEFAULT_TABLE table=rebalance_test_table_8 [id=f3134ee4e058445b955a09231a5e159f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:22.150983 31905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98e66cb7daff4977927911bbfad93b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:22.151203 31907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 930b5fcce62a4040a2e2b0f0f9462477. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:22.151904 31516 tablet_service.cc:1467] Processing CreateTablet for tablet 930b5fcce62a4040a2e2b0f0f9462477 (DEFAULT_TABLE table=rebalance_test_table_8 [id=f3134ee4e058445b955a09231a5e159f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:22.152276 31516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 930b5fcce62a4040a2e2b0f0f9462477. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:22.153818 31646 tablet_service.cc:1467] Processing CreateTablet for tablet 930b5fcce62a4040a2e2b0f0f9462477 (DEFAULT_TABLE table=rebalance_test_table_8 [id=f3134ee4e058445b955a09231a5e159f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:22.154209 31646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 930b5fcce62a4040a2e2b0f0f9462477. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:22.156759 31645 tablet_service.cc:1467] Processing CreateTablet for tablet 656e698c34394c38a723230d788b702a (DEFAULT_TABLE table=rebalance_test_table_8 [id=f3134ee4e058445b955a09231a5e159f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:22.157104 31645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 656e698c34394c38a723230d788b702a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:22.161350 31775 tablet_service.cc:1467] Processing CreateTablet for tablet 98e66cb7daff4977927911bbfad93b62 (DEFAULT_TABLE table=rebalance_test_table_8 [id=f3134ee4e058445b955a09231a5e159f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:22.161880 31775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98e66cb7daff4977927911bbfad93b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:22.170615 31646 tablet_service.cc:1467] Processing CreateTablet for tablet 98e66cb7daff4977927911bbfad93b62 (DEFAULT_TABLE table=rebalance_test_table_8 [id=f3134ee4e058445b955a09231a5e159f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:22.171094 31646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98e66cb7daff4977927911bbfad93b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:22.179042 734 tablet_bootstrap.cc:492] T 656e698c34394c38a723230d788b702a P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:22.179888 735 tablet_bootstrap.cc:492] T 930b5fcce62a4040a2e2b0f0f9462477 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:22.180756 734 tablet_bootstrap.cc:654] T 656e698c34394c38a723230d788b702a P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:22.181891 733 tablet_bootstrap.cc:492] T 656e698c34394c38a723230d788b702a P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:22.182574 736 tablet_bootstrap.cc:492] T 656e698c34394c38a723230d788b702a P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:22.183476 733 tablet_bootstrap.cc:654] T 656e698c34394c38a723230d788b702a P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:22.184046 735 tablet_bootstrap.cc:654] T 930b5fcce62a4040a2e2b0f0f9462477 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:22.185704 733 tablet_bootstrap.cc:492] T 656e698c34394c38a723230d788b702a P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:22.185858 736 tablet_bootstrap.cc:654] T 656e698c34394c38a723230d788b702a P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:22.185859 734 tablet_bootstrap.cc:492] T 656e698c34394c38a723230d788b702a P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:22.185889 733 ts_tablet_manager.cc:1367] T 656e698c34394c38a723230d788b702a P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:26:22.185976 734 ts_tablet_manager.cc:1367] T 656e698c34394c38a723230d788b702a P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 21:26:22.186717 733 raft_consensus.cc:348] T 656e698c34394c38a723230d788b702a P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:22.186842 734 raft_consensus.cc:348] T 656e698c34394c38a723230d788b702a P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:22.186959 733 raft_consensus.cc:374] T 656e698c34394c38a723230d788b702a P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:22.187191 734 raft_consensus.cc:374] T 656e698c34394c38a723230d788b702a P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:22.187238 733 raft_consensus.cc:729] T 656e698c34394c38a723230d788b702a P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:22.187368 734 raft_consensus.cc:729] T 656e698c34394c38a723230d788b702a P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:22.187500 733 consensus_queue.cc:260] T 656e698c34394c38a723230d788b702a P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:22.187683 734 consensus_queue.cc:260] T 656e698c34394c38a723230d788b702a P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:22.188566 734 ts_tablet_manager.cc:1398] T 656e698c34394c38a723230d788b702a P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:22.188915 733 ts_tablet_manager.cc:1398] T 656e698c34394c38a723230d788b702a P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:22.188927 734 tablet_bootstrap.cc:492] T 930b5fcce62a4040a2e2b0f0f9462477 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:22.191046 734 tablet_bootstrap.cc:654] T 930b5fcce62a4040a2e2b0f0f9462477 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:22.195565 736 tablet_bootstrap.cc:492] T 656e698c34394c38a723230d788b702a P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:22.195731 736 ts_tablet_manager.cc:1367] T 656e698c34394c38a723230d788b702a P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.004s
I20240502 21:26:22.196668 736 raft_consensus.cc:348] T 656e698c34394c38a723230d788b702a P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:22.196892 736 raft_consensus.cc:374] T 656e698c34394c38a723230d788b702a P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:22.196977 736 raft_consensus.cc:729] T 656e698c34394c38a723230d788b702a P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:22.197207 736 consensus_queue.cc:260] T 656e698c34394c38a723230d788b702a P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:22.198588 735 tablet_bootstrap.cc:492] T 930b5fcce62a4040a2e2b0f0f9462477 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:22.198768 735 ts_tablet_manager.cc:1367] T 930b5fcce62a4040a2e2b0f0f9462477 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.005s
I20240502 21:26:22.199163 734 tablet_bootstrap.cc:492] T 930b5fcce62a4040a2e2b0f0f9462477 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:22.199311 734 ts_tablet_manager.cc:1367] T 930b5fcce62a4040a2e2b0f0f9462477 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240502 21:26:22.199612 735 raft_consensus.cc:348] T 930b5fcce62a4040a2e2b0f0f9462477 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:22.199802 735 raft_consensus.cc:374] T 930b5fcce62a4040a2e2b0f0f9462477 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:22.199903 735 raft_consensus.cc:729] T 930b5fcce62a4040a2e2b0f0f9462477 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:22.199968 734 raft_consensus.cc:348] T 930b5fcce62a4040a2e2b0f0f9462477 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:22.200156 734 raft_consensus.cc:374] T 930b5fcce62a4040a2e2b0f0f9462477 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:22.200106 735 consensus_queue.cc:260] T 930b5fcce62a4040a2e2b0f0f9462477 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:22.200222 734 raft_consensus.cc:729] T 930b5fcce62a4040a2e2b0f0f9462477 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:22.200459 734 consensus_queue.cc:260] T 930b5fcce62a4040a2e2b0f0f9462477 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:22.200623 735 ts_tablet_manager.cc:1398] T 930b5fcce62a4040a2e2b0f0f9462477 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:22.200945 736 ts_tablet_manager.cc:1398] T 656e698c34394c38a723230d788b702a P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:26:22.201012 734 ts_tablet_manager.cc:1398] T 930b5fcce62a4040a2e2b0f0f9462477 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:22.201206 736 tablet_bootstrap.cc:492] T 930b5fcce62a4040a2e2b0f0f9462477 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:22.201234 734 tablet_bootstrap.cc:492] T 98e66cb7daff4977927911bbfad93b62 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:22.202692 734 tablet_bootstrap.cc:654] T 98e66cb7daff4977927911bbfad93b62 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:22.202704 736 tablet_bootstrap.cc:654] T 930b5fcce62a4040a2e2b0f0f9462477 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:22.205406 735 tablet_bootstrap.cc:492] T 98e66cb7daff4977927911bbfad93b62 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:22.207222 735 tablet_bootstrap.cc:654] T 98e66cb7daff4977927911bbfad93b62 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:22.208886 733 tablet_bootstrap.cc:492] T 98e66cb7daff4977927911bbfad93b62 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:22.209945 736 tablet_bootstrap.cc:492] T 930b5fcce62a4040a2e2b0f0f9462477 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:22.210155 736 ts_tablet_manager.cc:1367] T 930b5fcce62a4040a2e2b0f0f9462477 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20240502 21:26:22.210705 733 tablet_bootstrap.cc:654] T 98e66cb7daff4977927911bbfad93b62 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:22.211007 736 raft_consensus.cc:348] T 930b5fcce62a4040a2e2b0f0f9462477 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:22.211268 736 raft_consensus.cc:374] T 930b5fcce62a4040a2e2b0f0f9462477 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:22.211352 736 raft_consensus.cc:729] T 930b5fcce62a4040a2e2b0f0f9462477 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:22.211763 736 consensus_queue.cc:260] T 930b5fcce62a4040a2e2b0f0f9462477 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:22.212651 736 ts_tablet_manager.cc:1398] T 930b5fcce62a4040a2e2b0f0f9462477 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240502 21:26:22.213650 734 tablet_bootstrap.cc:492] T 98e66cb7daff4977927911bbfad93b62 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:22.213701 735 tablet_bootstrap.cc:492] T 98e66cb7daff4977927911bbfad93b62 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:22.213819 735 ts_tablet_manager.cc:1367] T 98e66cb7daff4977927911bbfad93b62 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.001s
I20240502 21:26:22.213775 734 ts_tablet_manager.cc:1367] T 98e66cb7daff4977927911bbfad93b62 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.008s
I20240502 21:26:22.214463 735 raft_consensus.cc:348] T 98e66cb7daff4977927911bbfad93b62 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:22.214522 734 raft_consensus.cc:348] T 98e66cb7daff4977927911bbfad93b62 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:22.214659 735 raft_consensus.cc:374] T 98e66cb7daff4977927911bbfad93b62 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:22.214676 734 raft_consensus.cc:374] T 98e66cb7daff4977927911bbfad93b62 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:22.214721 735 raft_consensus.cc:729] T 98e66cb7daff4977927911bbfad93b62 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:22.214756 734 raft_consensus.cc:729] T 98e66cb7daff4977927911bbfad93b62 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:22.214931 735 consensus_queue.cc:260] T 98e66cb7daff4977927911bbfad93b62 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:22.214963 734 consensus_queue.cc:260] T 98e66cb7daff4977927911bbfad93b62 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:22.215385 735 ts_tablet_manager.cc:1398] T 98e66cb7daff4977927911bbfad93b62 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:26:22.215498 734 ts_tablet_manager.cc:1398] T 98e66cb7daff4977927911bbfad93b62 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 21:26:22.216864 733 tablet_bootstrap.cc:492] T 98e66cb7daff4977927911bbfad93b62 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:22.217010 733 ts_tablet_manager.cc:1367] T 98e66cb7daff4977927911bbfad93b62 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 21:26:22.217573 733 raft_consensus.cc:348] T 98e66cb7daff4977927911bbfad93b62 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:22.217794 733 raft_consensus.cc:374] T 98e66cb7daff4977927911bbfad93b62 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:22.217870 733 raft_consensus.cc:729] T 98e66cb7daff4977927911bbfad93b62 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:22.218118 733 consensus_queue.cc:260] T 98e66cb7daff4977927911bbfad93b62 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:22.218626 733 ts_tablet_manager.cc:1398] T 98e66cb7daff4977927911bbfad93b62 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:22.295979 719 raft_consensus.cc:482] T 98e66cb7daff4977927911bbfad93b62 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:22.296108 719 raft_consensus.cc:504] T 98e66cb7daff4977927911bbfad93b62 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:22.296689 719 leader_election.cc:288] T 98e66cb7daff4977927911bbfad93b62 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:22.297003 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e66cb7daff4977927911bbfad93b62" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:22.297055 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e66cb7daff4977927911bbfad93b62" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:22.297188 31926 raft_consensus.cc:2454] T 98e66cb7daff4977927911bbfad93b62 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:22.297255 31665 raft_consensus.cc:2454] T 98e66cb7daff4977927911bbfad93b62 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:22.297581 31733 leader_election.cc:302] T 98e66cb7daff4977927911bbfad93b62 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:22.297825 719 raft_consensus.cc:2781] T 98e66cb7daff4977927911bbfad93b62 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:22.297928 719 raft_consensus.cc:482] T 98e66cb7daff4977927911bbfad93b62 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:22.298012 719 raft_consensus.cc:3037] T 98e66cb7daff4977927911bbfad93b62 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:22.299700 719 raft_consensus.cc:504] T 98e66cb7daff4977927911bbfad93b62 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:22.300141 719 leader_election.cc:288] T 98e66cb7daff4977927911bbfad93b62 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Requested vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:22.300514 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e66cb7daff4977927911bbfad93b62" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:22.300518 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e66cb7daff4977927911bbfad93b62" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:22.300666 31665 raft_consensus.cc:3037] T 98e66cb7daff4977927911bbfad93b62 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:22.300673 31926 raft_consensus.cc:3037] T 98e66cb7daff4977927911bbfad93b62 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:22.302448 31926 raft_consensus.cc:2454] T 98e66cb7daff4977927911bbfad93b62 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:22.302521 31665 raft_consensus.cc:2454] T 98e66cb7daff4977927911bbfad93b62 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:22.302888 31733 leader_election.cc:302] T 98e66cb7daff4977927911bbfad93b62 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 08d7a0680d244b55b93d17f9ab57b84c; no voters:
I20240502 21:26:22.303233 719 raft_consensus.cc:2781] T 98e66cb7daff4977927911bbfad93b62 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:22.303541 719 raft_consensus.cc:686] T 98e66cb7daff4977927911bbfad93b62 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Becoming Leader. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Running, Role: LEADER
I20240502 21:26:22.303766 719 consensus_queue.cc:237] T 98e66cb7daff4977927911bbfad93b62 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:22.305984 31389 catalog_manager.cc:5484] T 98e66cb7daff4977927911bbfad93b62 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: term changed from 0 to 1, leader changed from <none> to 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131). New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:22.307502 719 raft_consensus.cc:482] T 656e698c34394c38a723230d788b702a P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:22.307634 719 raft_consensus.cc:504] T 656e698c34394c38a723230d788b702a P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:22.308116 719 leader_election.cc:288] T 656e698c34394c38a723230d788b702a P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:22.308562 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "656e698c34394c38a723230d788b702a" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:22.308642 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "656e698c34394c38a723230d788b702a" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:22.308750 31665 raft_consensus.cc:2454] T 656e698c34394c38a723230d788b702a P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:22.308851 31535 raft_consensus.cc:2454] T 656e698c34394c38a723230d788b702a P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:22.309270 31733 leader_election.cc:302] T 656e698c34394c38a723230d788b702a P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:22.309531 719 raft_consensus.cc:2781] T 656e698c34394c38a723230d788b702a P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:22.309615 719 raft_consensus.cc:482] T 656e698c34394c38a723230d788b702a P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:22.309715 719 raft_consensus.cc:3037] T 656e698c34394c38a723230d788b702a P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:22.311393 719 raft_consensus.cc:504] T 656e698c34394c38a723230d788b702a P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:22.311893 719 leader_election.cc:288] T 656e698c34394c38a723230d788b702a P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Requested vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:22.312191 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "656e698c34394c38a723230d788b702a" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:26:22.312414 31535 raft_consensus.cc:3037] T 656e698c34394c38a723230d788b702a P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:22.312356 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "656e698c34394c38a723230d788b702a" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:22.312561 31665 raft_consensus.cc:3037] T 656e698c34394c38a723230d788b702a P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:22.315025 31665 raft_consensus.cc:2454] T 656e698c34394c38a723230d788b702a P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:22.315053 31535 raft_consensus.cc:2454] T 656e698c34394c38a723230d788b702a P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:22.315434 31733 leader_election.cc:302] T 656e698c34394c38a723230d788b702a P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:22.315652 719 raft_consensus.cc:2781] T 656e698c34394c38a723230d788b702a P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:22.315810 719 raft_consensus.cc:686] T 656e698c34394c38a723230d788b702a P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Becoming Leader. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Running, Role: LEADER
I20240502 21:26:22.316061 719 consensus_queue.cc:237] T 656e698c34394c38a723230d788b702a P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:22.318367 31389 catalog_manager.cc:5484] T 656e698c34394c38a723230d788b702a P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: term changed from 0 to 1, leader changed from <none> to 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131). New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:22.343130 728 raft_consensus.cc:482] T 930b5fcce62a4040a2e2b0f0f9462477 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:22.343286 728 raft_consensus.cc:504] T 930b5fcce62a4040a2e2b0f0f9462477 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:22.343744 728 leader_election.cc:288] T 930b5fcce62a4040a2e2b0f0f9462477 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:22.344143 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930b5fcce62a4040a2e2b0f0f9462477" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:22.344208 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930b5fcce62a4040a2e2b0f0f9462477" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:22.344355 31665 raft_consensus.cc:2454] T 930b5fcce62a4040a2e2b0f0f9462477 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:22.344378 31926 raft_consensus.cc:2454] T 930b5fcce62a4040a2e2b0f0f9462477 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:22.344707 31472 leader_election.cc:302] T 930b5fcce62a4040a2e2b0f0f9462477 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:22.344952 728 raft_consensus.cc:2781] T 930b5fcce62a4040a2e2b0f0f9462477 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:22.345055 728 raft_consensus.cc:482] T 930b5fcce62a4040a2e2b0f0f9462477 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:22.345139 728 raft_consensus.cc:3037] T 930b5fcce62a4040a2e2b0f0f9462477 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:22.346699 728 raft_consensus.cc:504] T 930b5fcce62a4040a2e2b0f0f9462477 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:22.347177 728 leader_election.cc:288] T 930b5fcce62a4040a2e2b0f0f9462477 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Requested vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:22.347537 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930b5fcce62a4040a2e2b0f0f9462477" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:22.347674 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930b5fcce62a4040a2e2b0f0f9462477" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:22.347745 31665 raft_consensus.cc:3037] T 930b5fcce62a4040a2e2b0f0f9462477 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:22.347818 31926 raft_consensus.cc:3037] T 930b5fcce62a4040a2e2b0f0f9462477 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:22.349756 31926 raft_consensus.cc:2454] T 930b5fcce62a4040a2e2b0f0f9462477 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:22.349854 31665 raft_consensus.cc:2454] T 930b5fcce62a4040a2e2b0f0f9462477 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:22.350127 31472 leader_election.cc:302] T 930b5fcce62a4040a2e2b0f0f9462477 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:22.350438 728 raft_consensus.cc:2781] T 930b5fcce62a4040a2e2b0f0f9462477 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:22.350772 728 raft_consensus.cc:686] T 930b5fcce62a4040a2e2b0f0f9462477 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:26:22.351017 728 consensus_queue.cc:237] T 930b5fcce62a4040a2e2b0f0f9462477 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:22.353576 31389 catalog_manager.cc:5484] T 930b5fcce62a4040a2e2b0f0f9462477 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 0 to 1, leader changed from <none> to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } }
W20240502 21:26:22.669358 751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:22.669644 751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:22.676388 751 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:26:22.699592 719 consensus_queue.cc:1035] T 656e698c34394c38a723230d788b702a P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:22.704475 746 consensus_queue.cc:1035] T 656e698c34394c38a723230d788b702a P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_8'
I20240502 21:26:22.807401 746 consensus_queue.cc:1035] T 98e66cb7daff4977927911bbfad93b62 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:22.815541 728 consensus_queue.cc:1035] T 930b5fcce62a4040a2e2b0f0f9462477 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:22.819211 31666 raft_consensus.cc:1261] T 930b5fcce62a4040a2e2b0f0f9462477 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:22.819715 728 consensus_queue.cc:1035] T 930b5fcce62a4040a2e2b0f0f9462477 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:22.820118 31926 raft_consensus.cc:1261] T 98e66cb7daff4977927911bbfad93b62 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:22.820883 746 consensus_queue.cc:1035] T 98e66cb7daff4977927911bbfad93b62 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 254.905 ms
time per row: 0.127452 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 21:26:23.935220 31390 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:23.945186 31516 tablet_service.cc:1467] Processing CreateTablet for tablet 988a0a9425434cd0b724710e9ebf23a6 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5eee55eb24e94c988b4ddbfe2f8442d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:23.945314 31905 tablet_service.cc:1467] Processing CreateTablet for tablet 988a0a9425434cd0b724710e9ebf23a6 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5eee55eb24e94c988b4ddbfe2f8442d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:23.945523 31775 tablet_service.cc:1467] Processing CreateTablet for tablet 988a0a9425434cd0b724710e9ebf23a6 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5eee55eb24e94c988b4ddbfe2f8442d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:23.945663 31516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 988a0a9425434cd0b724710e9ebf23a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:23.945711 31905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 988a0a9425434cd0b724710e9ebf23a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:23.945883 31775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 988a0a9425434cd0b724710e9ebf23a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:23.949891 822 tablet_bootstrap.cc:492] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:23.950836 31645 tablet_service.cc:1467] Processing CreateTablet for tablet fc93b5adc1d94e809e97e56b56822611 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5eee55eb24e94c988b4ddbfe2f8442d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:23.951229 31645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc93b5adc1d94e809e97e56b56822611. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:23.951309 31777 tablet_service.cc:1467] Processing CreateTablet for tablet 1ed225bdefc64ddd8eef2109bfd969df (DEFAULT_TABLE table=rebalance_test_table_9 [id=5eee55eb24e94c988b4ddbfe2f8442d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:23.951306 31515 tablet_service.cc:1467] Processing CreateTablet for tablet fc93b5adc1d94e809e97e56b56822611 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5eee55eb24e94c988b4ddbfe2f8442d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:23.951396 31514 tablet_service.cc:1467] Processing CreateTablet for tablet 1ed225bdefc64ddd8eef2109bfd969df (DEFAULT_TABLE table=rebalance_test_table_9 [id=5eee55eb24e94c988b4ddbfe2f8442d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:23.951653 31515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc93b5adc1d94e809e97e56b56822611. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:23.951645 31777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ed225bdefc64ddd8eef2109bfd969df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:23.952014 31776 tablet_service.cc:1467] Processing CreateTablet for tablet fc93b5adc1d94e809e97e56b56822611 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5eee55eb24e94c988b4ddbfe2f8442d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:23.952365 31776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc93b5adc1d94e809e97e56b56822611. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:23.956782 31514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ed225bdefc64ddd8eef2109bfd969df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:23.956835 31907 tablet_service.cc:1467] Processing CreateTablet for tablet 1ed225bdefc64ddd8eef2109bfd969df (DEFAULT_TABLE table=rebalance_test_table_9 [id=5eee55eb24e94c988b4ddbfe2f8442d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:23.957127 31907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ed225bdefc64ddd8eef2109bfd969df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:23.957702 822 tablet_bootstrap.cc:654] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:23.966282 826 tablet_bootstrap.cc:492] T fc93b5adc1d94e809e97e56b56822611 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:23.966344 825 tablet_bootstrap.cc:492] T 1ed225bdefc64ddd8eef2109bfd969df P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:23.966760 824 tablet_bootstrap.cc:492] T 1ed225bdefc64ddd8eef2109bfd969df P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:23.967849 826 tablet_bootstrap.cc:654] T fc93b5adc1d94e809e97e56b56822611 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:23.968037 822 tablet_bootstrap.cc:492] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:23.968209 822 ts_tablet_manager.cc:1367] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.006s
I20240502 21:26:23.968351 824 tablet_bootstrap.cc:654] T 1ed225bdefc64ddd8eef2109bfd969df P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:23.969120 822 raft_consensus.cc:348] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:23.969336 822 raft_consensus.cc:374] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:23.969435 822 raft_consensus.cc:729] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:23.969678 822 consensus_queue.cc:260] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:23.970295 822 ts_tablet_manager.cc:1398] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:23.970582 822 tablet_bootstrap.cc:492] T fc93b5adc1d94e809e97e56b56822611 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:23.971320 826 tablet_bootstrap.cc:492] T fc93b5adc1d94e809e97e56b56822611 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:23.971506 826 ts_tablet_manager.cc:1367] T fc93b5adc1d94e809e97e56b56822611 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240502 21:26:23.971920 822 tablet_bootstrap.cc:654] T fc93b5adc1d94e809e97e56b56822611 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:23.972404 826 raft_consensus.cc:348] T fc93b5adc1d94e809e97e56b56822611 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:23.972791 826 raft_consensus.cc:374] T fc93b5adc1d94e809e97e56b56822611 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:23.972930 826 raft_consensus.cc:729] T fc93b5adc1d94e809e97e56b56822611 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:23.972995 825 tablet_bootstrap.cc:654] T 1ed225bdefc64ddd8eef2109bfd969df P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:23.973534 826 consensus_queue.cc:260] T fc93b5adc1d94e809e97e56b56822611 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:23.973882 817 raft_consensus.cc:482] T fc93b5adc1d94e809e97e56b56822611 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:23.974042 817 raft_consensus.cc:504] T fc93b5adc1d94e809e97e56b56822611 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:23.974114 826 ts_tablet_manager.cc:1398] T fc93b5adc1d94e809e97e56b56822611 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:23.974647 817 leader_election.cc:288] T fc93b5adc1d94e809e97e56b56822611 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:23.975244 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc93b5adc1d94e809e97e56b56822611" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:23.975708 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc93b5adc1d94e809e97e56b56822611" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
W20240502 21:26:23.976713 31603 leader_election.cc:341] T fc93b5adc1d94e809e97e56b56822611 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:26:23.977022 31602 leader_election.cc:341] T fc93b5adc1d94e809e97e56b56822611 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:23.977180 31602 leader_election.cc:302] T fc93b5adc1d94e809e97e56b56822611 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada; no voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f
I20240502 21:26:23.977639 509 raft_consensus.cc:2726] T fc93b5adc1d94e809e97e56b56822611 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:26:23.978740 824 tablet_bootstrap.cc:492] T 1ed225bdefc64ddd8eef2109bfd969df P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:23.978870 824 ts_tablet_manager.cc:1367] T 1ed225bdefc64ddd8eef2109bfd969df P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20240502 21:26:23.979779 824 raft_consensus.cc:348] T 1ed225bdefc64ddd8eef2109bfd969df P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:23.980055 824 raft_consensus.cc:374] T 1ed225bdefc64ddd8eef2109bfd969df P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:23.980147 824 raft_consensus.cc:729] T 1ed225bdefc64ddd8eef2109bfd969df P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:23.980405 824 consensus_queue.cc:260] T 1ed225bdefc64ddd8eef2109bfd969df P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:23.980949 824 ts_tablet_manager.cc:1398] T 1ed225bdefc64ddd8eef2109bfd969df P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:23.981218 824 tablet_bootstrap.cc:492] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:23.982622 824 tablet_bootstrap.cc:654] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:23.984184 824 tablet_bootstrap.cc:492] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:23.984457 824 ts_tablet_manager.cc:1367] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 21:26:23.985046 824 raft_consensus.cc:348] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:23.985211 824 raft_consensus.cc:374] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:23.985283 824 raft_consensus.cc:729] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:23.985491 824 consensus_queue.cc:260] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:23.985965 824 ts_tablet_manager.cc:1398] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 21:26:23.986198 824 tablet_bootstrap.cc:492] T 988a0a9425434cd0b724710e9ebf23a6 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:23.987277 824 tablet_bootstrap.cc:654] T 988a0a9425434cd0b724710e9ebf23a6 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:23.988657 824 tablet_bootstrap.cc:492] T 988a0a9425434cd0b724710e9ebf23a6 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:23.988787 824 ts_tablet_manager.cc:1367] T 988a0a9425434cd0b724710e9ebf23a6 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:26:23.989341 824 raft_consensus.cc:348] T 988a0a9425434cd0b724710e9ebf23a6 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:23.989475 824 raft_consensus.cc:374] T 988a0a9425434cd0b724710e9ebf23a6 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:23.989562 824 raft_consensus.cc:729] T 988a0a9425434cd0b724710e9ebf23a6 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:23.989750 824 consensus_queue.cc:260] T 988a0a9425434cd0b724710e9ebf23a6 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:23.990279 824 ts_tablet_manager.cc:1398] T 988a0a9425434cd0b724710e9ebf23a6 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 21:26:23.992998 822 tablet_bootstrap.cc:492] T fc93b5adc1d94e809e97e56b56822611 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:23.993156 822 ts_tablet_manager.cc:1367] T fc93b5adc1d94e809e97e56b56822611 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20240502 21:26:23.993307 825 tablet_bootstrap.cc:492] T 1ed225bdefc64ddd8eef2109bfd969df P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:23.993467 825 ts_tablet_manager.cc:1367] T 1ed225bdefc64ddd8eef2109bfd969df P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20240502 21:26:23.993719 822 raft_consensus.cc:348] T fc93b5adc1d94e809e97e56b56822611 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:23.993880 822 raft_consensus.cc:374] T fc93b5adc1d94e809e97e56b56822611 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:23.993952 822 raft_consensus.cc:729] T fc93b5adc1d94e809e97e56b56822611 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:23.994204 822 consensus_queue.cc:260] T fc93b5adc1d94e809e97e56b56822611 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:23.994321 825 raft_consensus.cc:348] T 1ed225bdefc64ddd8eef2109bfd969df P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:23.994737 825 raft_consensus.cc:374] T 1ed225bdefc64ddd8eef2109bfd969df P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:23.994763 822 ts_tablet_manager.cc:1398] T fc93b5adc1d94e809e97e56b56822611 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:23.994839 825 raft_consensus.cc:729] T 1ed225bdefc64ddd8eef2109bfd969df P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:23.994999 822 tablet_bootstrap.cc:492] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:23.995045 825 consensus_queue.cc:260] T 1ed225bdefc64ddd8eef2109bfd969df P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:23.996356 825 ts_tablet_manager.cc:1398] T 1ed225bdefc64ddd8eef2109bfd969df P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:23.996651 825 tablet_bootstrap.cc:492] T 988a0a9425434cd0b724710e9ebf23a6 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:23.996760 822 tablet_bootstrap.cc:654] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:23.997725 825 tablet_bootstrap.cc:654] T 988a0a9425434cd0b724710e9ebf23a6 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:24.000526 825 tablet_bootstrap.cc:492] T 988a0a9425434cd0b724710e9ebf23a6 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:24.000720 825 ts_tablet_manager.cc:1367] T 988a0a9425434cd0b724710e9ebf23a6 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:26:24.001308 825 raft_consensus.cc:348] T 988a0a9425434cd0b724710e9ebf23a6 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:24.001469 825 raft_consensus.cc:374] T 988a0a9425434cd0b724710e9ebf23a6 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:24.001583 825 raft_consensus.cc:729] T 988a0a9425434cd0b724710e9ebf23a6 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:24.001808 825 consensus_queue.cc:260] T 988a0a9425434cd0b724710e9ebf23a6 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:24.002394 825 ts_tablet_manager.cc:1398] T 988a0a9425434cd0b724710e9ebf23a6 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:24.004333 822 tablet_bootstrap.cc:492] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:24.004472 822 ts_tablet_manager.cc:1367] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.001s
I20240502 21:26:24.005048 822 raft_consensus.cc:348] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:24.005223 822 raft_consensus.cc:374] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:24.005312 822 raft_consensus.cc:729] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:24.005496 822 consensus_queue.cc:260] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:24.006055 822 ts_tablet_manager.cc:1398] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 21:26:24.008761 728 raft_consensus.cc:482] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:24.008872 728 raft_consensus.cc:504] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:24.009464 728 leader_election.cc:288] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:24.009802 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ed225bdefc64ddd8eef2109bfd969df" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:24.009833 31927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ed225bdefc64ddd8eef2109bfd969df" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:24.010007 31797 raft_consensus.cc:2454] T 1ed225bdefc64ddd8eef2109bfd969df P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:24.010041 31927 raft_consensus.cc:2454] T 1ed225bdefc64ddd8eef2109bfd969df P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:24.010362 31471 leader_election.cc:302] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:24.010671 728 raft_consensus.cc:2781] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:24.010777 728 raft_consensus.cc:482] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:24.010860 728 raft_consensus.cc:3037] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:24.012603 728 raft_consensus.cc:504] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:24.013057 728 leader_election.cc:288] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Requested vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:24.013475 31927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ed225bdefc64ddd8eef2109bfd969df" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:24.013463 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ed225bdefc64ddd8eef2109bfd969df" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:24.013653 31797 raft_consensus.cc:3037] T 1ed225bdefc64ddd8eef2109bfd969df P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:24.013653 31927 raft_consensus.cc:3037] T 1ed225bdefc64ddd8eef2109bfd969df P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:24.016113 31927 raft_consensus.cc:2454] T 1ed225bdefc64ddd8eef2109bfd969df P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:24.016113 31797 raft_consensus.cc:2454] T 1ed225bdefc64ddd8eef2109bfd969df P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:24.016479 31472 leader_election.cc:302] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:24.016716 728 raft_consensus.cc:2781] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:24.017040 728 raft_consensus.cc:686] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:26:24.017287 728 consensus_queue.cc:237] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:24.019620 31390 catalog_manager.cc:5484] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 0 to 1, leader changed from <none> to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:24.147048 719 raft_consensus.cc:482] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:24.147262 719 raft_consensus.cc:504] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:24.147936 719 leader_election.cc:288] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:24.148218 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc93b5adc1d94e809e97e56b56822611" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:24.148350 31667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc93b5adc1d94e809e97e56b56822611" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:24.148463 31535 raft_consensus.cc:2454] T fc93b5adc1d94e809e97e56b56822611 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:24.148610 31667 raft_consensus.cc:2454] T fc93b5adc1d94e809e97e56b56822611 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:24.148830 31733 leader_election.cc:302] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:24.148891 728 raft_consensus.cc:482] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:24.149049 728 raft_consensus.cc:504] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:24.149338 719 raft_consensus.cc:2781] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:24.149444 719 raft_consensus.cc:482] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:24.149559 719 raft_consensus.cc:3037] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:24.150110 728 leader_election.cc:288] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:24.150030 31927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "988a0a9425434cd0b724710e9ebf23a6" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:24.150350 31927 raft_consensus.cc:2454] T 988a0a9425434cd0b724710e9ebf23a6 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:24.150621 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "988a0a9425434cd0b724710e9ebf23a6" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:24.150694 31472 leader_election.cc:302] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:24.150856 31797 raft_consensus.cc:2454] T 988a0a9425434cd0b724710e9ebf23a6 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:24.150978 728 raft_consensus.cc:2781] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:24.151064 728 raft_consensus.cc:482] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:24.151521 728 raft_consensus.cc:3037] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:24.152472 719 raft_consensus.cc:504] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:24.152987 719 leader_election.cc:288] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Requested vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:24.153374 31667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc93b5adc1d94e809e97e56b56822611" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:24.153390 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc93b5adc1d94e809e97e56b56822611" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:26:24.153568 31667 raft_consensus.cc:3037] T fc93b5adc1d94e809e97e56b56822611 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:24.153604 31535 raft_consensus.cc:3037] T fc93b5adc1d94e809e97e56b56822611 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:24.153774 728 raft_consensus.cc:504] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:24.154632 31927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "988a0a9425434cd0b724710e9ebf23a6" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:24.154727 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "988a0a9425434cd0b724710e9ebf23a6" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:24.154892 31797 raft_consensus.cc:3037] T 988a0a9425434cd0b724710e9ebf23a6 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:24.154901 31927 raft_consensus.cc:3037] T 988a0a9425434cd0b724710e9ebf23a6 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:24.155885 31667 raft_consensus.cc:2454] T fc93b5adc1d94e809e97e56b56822611 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:24.156124 31535 raft_consensus.cc:2454] T fc93b5adc1d94e809e97e56b56822611 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:24.156296 31733 leader_election.cc:302] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 08d7a0680d244b55b93d17f9ab57b84c; no voters:
I20240502 21:26:24.156499 728 leader_election.cc:288] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Requested vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:24.156731 719 raft_consensus.cc:2781] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:24.157204 719 raft_consensus.cc:686] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Becoming Leader. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Running, Role: LEADER
I20240502 21:26:24.157459 31797 raft_consensus.cc:2454] T 988a0a9425434cd0b724710e9ebf23a6 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:24.157541 719 consensus_queue.cc:237] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:24.157881 31471 leader_election.cc:302] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:24.158104 728 raft_consensus.cc:2781] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:24.158273 728 raft_consensus.cc:686] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:26:24.158324 31927 raft_consensus.cc:2454] T 988a0a9425434cd0b724710e9ebf23a6 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:24.158510 728 consensus_queue.cc:237] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:24.162570 31390 catalog_manager.cc:5484] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: term changed from 0 to 1, leader changed from <none> to 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131). New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:24.162750 31394 catalog_manager.cc:5484] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 0 to 1, leader changed from <none> to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:24.459041 728 consensus_queue.cc:1035] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:24.464488 728 consensus_queue.cc:1035] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:24.510954 841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:24.511226 841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:24.518152 841 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240502 21:26:24.589653 728 consensus_queue.cc:1035] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:24.595348 835 consensus_queue.cc:1035] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:24.610841 719 consensus_queue.cc:1035] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:24.620249 837 consensus_queue.cc:1035] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 255.757 ms
time per row: 0.127879 ms
I20240502 21:26:24.917517 31393 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:24.926012 31514 tablet_service.cc:1467] Processing CreateTablet for tablet 900be14c60dc4be0adc22c2bcee4dd81 (DEFAULT_TABLE table=rebalance_test_table_10 [id=ff1e1957a03340f382098859ea2c90a8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:24.926004 31645 tablet_service.cc:1467] Processing CreateTablet for tablet 900be14c60dc4be0adc22c2bcee4dd81 (DEFAULT_TABLE table=rebalance_test_table_10 [id=ff1e1957a03340f382098859ea2c90a8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:24.926431 31645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 900be14c60dc4be0adc22c2bcee4dd81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:24.926445 31514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 900be14c60dc4be0adc22c2bcee4dd81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:24.926563 31907 tablet_service.cc:1467] Processing CreateTablet for tablet 900be14c60dc4be0adc22c2bcee4dd81 (DEFAULT_TABLE table=rebalance_test_table_10 [id=ff1e1957a03340f382098859ea2c90a8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:24.926935 31907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 900be14c60dc4be0adc22c2bcee4dd81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:24.928103 31647 tablet_service.cc:1467] Processing CreateTablet for tablet 0af9a3f71bcc47bd9290d4d75f65ca24 (DEFAULT_TABLE table=rebalance_test_table_10 [id=ff1e1957a03340f382098859ea2c90a8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:24.928404 31647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0af9a3f71bcc47bd9290d4d75f65ca24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:24.929055 31905 tablet_service.cc:1467] Processing CreateTablet for tablet 9a0d1c43a0794c2884dfef498f046f6d (DEFAULT_TABLE table=rebalance_test_table_10 [id=ff1e1957a03340f382098859ea2c90a8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:24.929083 31906 tablet_service.cc:1467] Processing CreateTablet for tablet 0af9a3f71bcc47bd9290d4d75f65ca24 (DEFAULT_TABLE table=rebalance_test_table_10 [id=ff1e1957a03340f382098859ea2c90a8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:24.929416 31905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a0d1c43a0794c2884dfef498f046f6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:24.932134 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0af9a3f71bcc47bd9290d4d75f65ca24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:24.932255 31516 tablet_service.cc:1467] Processing CreateTablet for tablet 0af9a3f71bcc47bd9290d4d75f65ca24 (DEFAULT_TABLE table=rebalance_test_table_10 [id=ff1e1957a03340f382098859ea2c90a8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:24.932284 31646 tablet_service.cc:1467] Processing CreateTablet for tablet 9a0d1c43a0794c2884dfef498f046f6d (DEFAULT_TABLE table=rebalance_test_table_10 [id=ff1e1957a03340f382098859ea2c90a8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:24.932642 31516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0af9a3f71bcc47bd9290d4d75f65ca24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:24.932658 31646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a0d1c43a0794c2884dfef498f046f6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:24.937288 31777 tablet_service.cc:1467] Processing CreateTablet for tablet 9a0d1c43a0794c2884dfef498f046f6d (DEFAULT_TABLE table=rebalance_test_table_10 [id=ff1e1957a03340f382098859ea2c90a8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:24.937695 31777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a0d1c43a0794c2884dfef498f046f6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:24.949553 906 tablet_bootstrap.cc:492] T 900be14c60dc4be0adc22c2bcee4dd81 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:24.950076 905 tablet_bootstrap.cc:492] T 900be14c60dc4be0adc22c2bcee4dd81 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:24.951808 905 tablet_bootstrap.cc:654] T 900be14c60dc4be0adc22c2bcee4dd81 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:24.951858 907 tablet_bootstrap.cc:492] T 9a0d1c43a0794c2884dfef498f046f6d P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:24.952062 906 tablet_bootstrap.cc:654] T 900be14c60dc4be0adc22c2bcee4dd81 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:24.952469 903 tablet_bootstrap.cc:492] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:24.954687 907 tablet_bootstrap.cc:654] T 9a0d1c43a0794c2884dfef498f046f6d P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:24.954918 903 tablet_bootstrap.cc:654] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:24.961500 906 tablet_bootstrap.cc:492] T 900be14c60dc4be0adc22c2bcee4dd81 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:24.961699 906 ts_tablet_manager.cc:1367] T 900be14c60dc4be0adc22c2bcee4dd81 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.008s
I20240502 21:26:24.962474 906 raft_consensus.cc:348] T 900be14c60dc4be0adc22c2bcee4dd81 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:24.962626 906 raft_consensus.cc:374] T 900be14c60dc4be0adc22c2bcee4dd81 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:24.962738 906 raft_consensus.cc:729] T 900be14c60dc4be0adc22c2bcee4dd81 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:24.962985 906 consensus_queue.cc:260] T 900be14c60dc4be0adc22c2bcee4dd81 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:24.963670 906 ts_tablet_manager.cc:1398] T 900be14c60dc4be0adc22c2bcee4dd81 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:26:24.964053 906 tablet_bootstrap.cc:492] T 9a0d1c43a0794c2884dfef498f046f6d P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:24.964493 907 tablet_bootstrap.cc:492] T 9a0d1c43a0794c2884dfef498f046f6d P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:24.964620 907 ts_tablet_manager.cc:1367] T 9a0d1c43a0794c2884dfef498f046f6d P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20240502 21:26:24.965231 903 tablet_bootstrap.cc:492] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:24.965358 903 ts_tablet_manager.cc:1367] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.004s
I20240502 21:26:24.965314 907 raft_consensus.cc:348] T 9a0d1c43a0794c2884dfef498f046f6d P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:24.965493 906 tablet_bootstrap.cc:654] T 9a0d1c43a0794c2884dfef498f046f6d P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:24.965519 907 raft_consensus.cc:374] T 9a0d1c43a0794c2884dfef498f046f6d P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:24.965596 907 raft_consensus.cc:729] T 9a0d1c43a0794c2884dfef498f046f6d P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:24.965821 907 consensus_queue.cc:260] T 9a0d1c43a0794c2884dfef498f046f6d P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:24.966053 903 raft_consensus.cc:348] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:24.966234 903 raft_consensus.cc:374] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:24.966310 903 raft_consensus.cc:729] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:24.966431 907 ts_tablet_manager.cc:1398] T 9a0d1c43a0794c2884dfef498f046f6d P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:24.966531 903 consensus_queue.cc:260] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:24.967859 903 ts_tablet_manager.cc:1398] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:24.968127 903 tablet_bootstrap.cc:492] T 9a0d1c43a0794c2884dfef498f046f6d P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:24.969848 906 tablet_bootstrap.cc:492] T 9a0d1c43a0794c2884dfef498f046f6d P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:24.970105 906 ts_tablet_manager.cc:1367] T 9a0d1c43a0794c2884dfef498f046f6d P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20240502 21:26:24.970774 906 raft_consensus.cc:348] T 9a0d1c43a0794c2884dfef498f046f6d P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:24.970932 906 raft_consensus.cc:374] T 9a0d1c43a0794c2884dfef498f046f6d P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:24.970986 903 tablet_bootstrap.cc:654] T 9a0d1c43a0794c2884dfef498f046f6d P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:24.971015 906 raft_consensus.cc:729] T 9a0d1c43a0794c2884dfef498f046f6d P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:24.971273 906 consensus_queue.cc:260] T 9a0d1c43a0794c2884dfef498f046f6d P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:24.971814 906 ts_tablet_manager.cc:1398] T 9a0d1c43a0794c2884dfef498f046f6d P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:24.972210 906 tablet_bootstrap.cc:492] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:24.972748 905 tablet_bootstrap.cc:492] T 900be14c60dc4be0adc22c2bcee4dd81 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:24.972941 905 ts_tablet_manager.cc:1367] T 900be14c60dc4be0adc22c2bcee4dd81 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.011s
I20240502 21:26:24.973374 906 tablet_bootstrap.cc:654] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:24.973838 905 raft_consensus.cc:348] T 900be14c60dc4be0adc22c2bcee4dd81 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:24.974078 905 raft_consensus.cc:374] T 900be14c60dc4be0adc22c2bcee4dd81 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:24.974195 905 raft_consensus.cc:729] T 900be14c60dc4be0adc22c2bcee4dd81 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:24.974481 905 consensus_queue.cc:260] T 900be14c60dc4be0adc22c2bcee4dd81 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:24.974771 906 tablet_bootstrap.cc:492] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:24.974886 906 ts_tablet_manager.cc:1367] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:24.975195 905 ts_tablet_manager.cc:1398] T 900be14c60dc4be0adc22c2bcee4dd81 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:26:24.975486 906 raft_consensus.cc:348] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:24.975646 906 raft_consensus.cc:374] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:24.975721 906 raft_consensus.cc:729] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:24.975920 906 consensus_queue.cc:260] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:24.976460 905 tablet_bootstrap.cc:492] T 0af9a3f71bcc47bd9290d4d75f65ca24 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:24.976550 906 ts_tablet_manager.cc:1398] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:24.977380 903 tablet_bootstrap.cc:492] T 9a0d1c43a0794c2884dfef498f046f6d P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:24.977530 903 ts_tablet_manager.cc:1367] T 9a0d1c43a0794c2884dfef498f046f6d P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240502 21:26:24.978183 903 raft_consensus.cc:348] T 9a0d1c43a0794c2884dfef498f046f6d P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:24.978366 903 raft_consensus.cc:374] T 9a0d1c43a0794c2884dfef498f046f6d P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:24.978439 903 raft_consensus.cc:729] T 9a0d1c43a0794c2884dfef498f046f6d P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:24.978631 903 consensus_queue.cc:260] T 9a0d1c43a0794c2884dfef498f046f6d P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:24.978914 905 tablet_bootstrap.cc:654] T 0af9a3f71bcc47bd9290d4d75f65ca24 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:24.979624 903 ts_tablet_manager.cc:1398] T 9a0d1c43a0794c2884dfef498f046f6d P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:24.979867 903 tablet_bootstrap.cc:492] T 900be14c60dc4be0adc22c2bcee4dd81 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:24.980779 905 tablet_bootstrap.cc:492] T 0af9a3f71bcc47bd9290d4d75f65ca24 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:24.980921 905 ts_tablet_manager.cc:1367] T 0af9a3f71bcc47bd9290d4d75f65ca24 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:26:24.981608 905 raft_consensus.cc:348] T 0af9a3f71bcc47bd9290d4d75f65ca24 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:24.981809 905 raft_consensus.cc:374] T 0af9a3f71bcc47bd9290d4d75f65ca24 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:24.981896 905 raft_consensus.cc:729] T 0af9a3f71bcc47bd9290d4d75f65ca24 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:24.981994 903 tablet_bootstrap.cc:654] T 900be14c60dc4be0adc22c2bcee4dd81 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:24.982149 905 consensus_queue.cc:260] T 0af9a3f71bcc47bd9290d4d75f65ca24 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:24.982590 905 ts_tablet_manager.cc:1398] T 0af9a3f71bcc47bd9290d4d75f65ca24 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:24.986346 903 tablet_bootstrap.cc:492] T 900be14c60dc4be0adc22c2bcee4dd81 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:24.986469 903 ts_tablet_manager.cc:1367] T 900be14c60dc4be0adc22c2bcee4dd81 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20240502 21:26:24.987032 903 raft_consensus.cc:348] T 900be14c60dc4be0adc22c2bcee4dd81 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:24.987198 903 raft_consensus.cc:374] T 900be14c60dc4be0adc22c2bcee4dd81 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:24.987260 903 raft_consensus.cc:729] T 900be14c60dc4be0adc22c2bcee4dd81 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:24.987442 903 consensus_queue.cc:260] T 900be14c60dc4be0adc22c2bcee4dd81 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:24.988258 903 ts_tablet_manager.cc:1398] T 900be14c60dc4be0adc22c2bcee4dd81 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:25.016407 885 raft_consensus.cc:482] T 0af9a3f71bcc47bd9290d4d75f65ca24 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:25.016597 885 raft_consensus.cc:504] T 0af9a3f71bcc47bd9290d4d75f65ca24 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:25.017346 885 leader_election.cc:288] T 0af9a3f71bcc47bd9290d4d75f65ca24 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:25.017639 31667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af9a3f71bcc47bd9290d4d75f65ca24" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:25.017810 31667 raft_consensus.cc:2454] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:25.017817 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af9a3f71bcc47bd9290d4d75f65ca24" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:25.018002 31926 raft_consensus.cc:2454] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:25.018141 31472 leader_election.cc:302] T 0af9a3f71bcc47bd9290d4d75f65ca24 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:25.018395 885 raft_consensus.cc:2781] T 0af9a3f71bcc47bd9290d4d75f65ca24 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:25.018527 885 raft_consensus.cc:482] T 0af9a3f71bcc47bd9290d4d75f65ca24 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:25.018608 885 raft_consensus.cc:3037] T 0af9a3f71bcc47bd9290d4d75f65ca24 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:25.020372 885 raft_consensus.cc:504] T 0af9a3f71bcc47bd9290d4d75f65ca24 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:25.020866 885 leader_election.cc:288] T 0af9a3f71bcc47bd9290d4d75f65ca24 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Requested vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:25.021283 31667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af9a3f71bcc47bd9290d4d75f65ca24" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:25.021325 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af9a3f71bcc47bd9290d4d75f65ca24" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:25.021484 31926 raft_consensus.cc:3037] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:25.021489 31667 raft_consensus.cc:3037] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:25.024232 31926 raft_consensus.cc:2454] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:25.024312 31667 raft_consensus.cc:2454] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:25.024631 31472 leader_election.cc:302] T 0af9a3f71bcc47bd9290d4d75f65ca24 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:25.024890 885 raft_consensus.cc:2781] T 0af9a3f71bcc47bd9290d4d75f65ca24 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:25.025038 885 raft_consensus.cc:686] T 0af9a3f71bcc47bd9290d4d75f65ca24 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:26:25.025307 885 consensus_queue.cc:237] T 0af9a3f71bcc47bd9290d4d75f65ca24 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:25.027814 31394 catalog_manager.cc:5484] T 0af9a3f71bcc47bd9290d4d75f65ca24 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 0 to 1, leader changed from <none> to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:25.052664 897 raft_consensus.cc:482] T 9a0d1c43a0794c2884dfef498f046f6d P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:25.052845 897 raft_consensus.cc:504] T 9a0d1c43a0794c2884dfef498f046f6d P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:25.053494 897 leader_election.cc:288] T 9a0d1c43a0794c2884dfef498f046f6d P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:25.053885 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0d1c43a0794c2884dfef498f046f6d" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:25.053997 31667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0d1c43a0794c2884dfef498f046f6d" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:25.054126 31926 raft_consensus.cc:2454] T 9a0d1c43a0794c2884dfef498f046f6d P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:25.054247 31667 raft_consensus.cc:2454] T 9a0d1c43a0794c2884dfef498f046f6d P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:25.054486 31733 leader_election.cc:302] T 9a0d1c43a0794c2884dfef498f046f6d P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:25.054697 897 raft_consensus.cc:2781] T 9a0d1c43a0794c2884dfef498f046f6d P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:25.054795 897 raft_consensus.cc:482] T 9a0d1c43a0794c2884dfef498f046f6d P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:25.054909 897 raft_consensus.cc:3037] T 9a0d1c43a0794c2884dfef498f046f6d P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:25.056890 897 raft_consensus.cc:504] T 9a0d1c43a0794c2884dfef498f046f6d P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:25.057309 897 leader_election.cc:288] T 9a0d1c43a0794c2884dfef498f046f6d P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Requested vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:25.057744 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0d1c43a0794c2884dfef498f046f6d" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:25.057818 31667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0d1c43a0794c2884dfef498f046f6d" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:25.057972 31926 raft_consensus.cc:3037] T 9a0d1c43a0794c2884dfef498f046f6d P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:25.058056 31667 raft_consensus.cc:3037] T 9a0d1c43a0794c2884dfef498f046f6d P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:25.060798 31926 raft_consensus.cc:2454] T 9a0d1c43a0794c2884dfef498f046f6d P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:25.060804 31667 raft_consensus.cc:2454] T 9a0d1c43a0794c2884dfef498f046f6d P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:25.061254 31733 leader_election.cc:302] T 9a0d1c43a0794c2884dfef498f046f6d P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 08d7a0680d244b55b93d17f9ab57b84c; no voters:
I20240502 21:26:25.061498 897 raft_consensus.cc:2781] T 9a0d1c43a0794c2884dfef498f046f6d P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:25.061655 897 raft_consensus.cc:686] T 9a0d1c43a0794c2884dfef498f046f6d P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Becoming Leader. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Running, Role: LEADER
I20240502 21:26:25.061924 897 consensus_queue.cc:237] T 9a0d1c43a0794c2884dfef498f046f6d P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:25.066036 31394 catalog_manager.cc:5484] T 9a0d1c43a0794c2884dfef498f046f6d P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: term changed from 0 to 1, leader changed from <none> to 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131). New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:25.108016 720 raft_consensus.cc:482] T 900be14c60dc4be0adc22c2bcee4dd81 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:25.108170 720 raft_consensus.cc:504] T 900be14c60dc4be0adc22c2bcee4dd81 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:25.108796 720 leader_election.cc:288] T 900be14c60dc4be0adc22c2bcee4dd81 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:25.109133 31667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "900be14c60dc4be0adc22c2bcee4dd81" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:25.109256 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "900be14c60dc4be0adc22c2bcee4dd81" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:25.109355 31667 raft_consensus.cc:2454] T 900be14c60dc4be0adc22c2bcee4dd81 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:25.109488 31535 raft_consensus.cc:2454] T 900be14c60dc4be0adc22c2bcee4dd81 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:25.109766 31863 leader_election.cc:302] T 900be14c60dc4be0adc22c2bcee4dd81 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:25.110000 720 raft_consensus.cc:2781] T 900be14c60dc4be0adc22c2bcee4dd81 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:25.110095 720 raft_consensus.cc:482] T 900be14c60dc4be0adc22c2bcee4dd81 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:25.110177 720 raft_consensus.cc:3037] T 900be14c60dc4be0adc22c2bcee4dd81 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:25.111922 720 raft_consensus.cc:504] T 900be14c60dc4be0adc22c2bcee4dd81 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:25.112376 720 leader_election.cc:288] T 900be14c60dc4be0adc22c2bcee4dd81 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Requested vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:25.112891 31667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "900be14c60dc4be0adc22c2bcee4dd81" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:25.112936 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "900be14c60dc4be0adc22c2bcee4dd81" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:26:25.113082 31667 raft_consensus.cc:3037] T 900be14c60dc4be0adc22c2bcee4dd81 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:25.113118 31535 raft_consensus.cc:3037] T 900be14c60dc4be0adc22c2bcee4dd81 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:25.115550 31667 raft_consensus.cc:2454] T 900be14c60dc4be0adc22c2bcee4dd81 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:26:25.115563 31535 raft_consensus.cc:2454] T 900be14c60dc4be0adc22c2bcee4dd81 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:26:25.115904 31863 leader_election.cc:302] T 900be14c60dc4be0adc22c2bcee4dd81 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:25.116119 720 raft_consensus.cc:2781] T 900be14c60dc4be0adc22c2bcee4dd81 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:25.116622 720 raft_consensus.cc:686] T 900be14c60dc4be0adc22c2bcee4dd81 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Running, Role: LEADER
I20240502 21:26:25.116928 720 consensus_queue.cc:237] T 900be14c60dc4be0adc22c2bcee4dd81 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:25.120286 31394 catalog_manager.cc:5484] T 900be14c60dc4be0adc22c2bcee4dd81 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132). New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } }
W20240502 21:26:25.430349 921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:25.430613 921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:25.437283 921 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240502 21:26:25.485476 885 consensus_queue.cc:1035] T 0af9a3f71bcc47bd9290d4d75f65ca24 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:25.485467 897 consensus_queue.cc:1035] T 9a0d1c43a0794c2884dfef498f046f6d P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:25.507594 885 consensus_queue.cc:1035] T 0af9a3f71bcc47bd9290d4d75f65ca24 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:25.561585 919 consensus_queue.cc:1035] T 900be14c60dc4be0adc22c2bcee4dd81 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:25.563567 897 consensus_queue.cc:1035] T 9a0d1c43a0794c2884dfef498f046f6d P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:25.590436 720 consensus_queue.cc:1035] T 900be14c60dc4be0adc22c2bcee4dd81 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 224.461 ms
time per row: 0.112231 ms
I20240502 21:26:25.820581 31389 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:25.829087 31777 tablet_service.cc:1467] Processing CreateTablet for tablet 55fc18e4cc98446ca50c1dd2b6a9a89d (DEFAULT_TABLE table=rebalance_test_table_11 [id=c46054f8169d4487b3c912c0681b1c3a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:25.829566 31777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55fc18e4cc98446ca50c1dd2b6a9a89d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:25.829665 31647 tablet_service.cc:1467] Processing CreateTablet for tablet 55fc18e4cc98446ca50c1dd2b6a9a89d (DEFAULT_TABLE table=rebalance_test_table_11 [id=c46054f8169d4487b3c912c0681b1c3a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:25.830076 31647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55fc18e4cc98446ca50c1dd2b6a9a89d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:25.830720 31907 tablet_service.cc:1467] Processing CreateTablet for tablet 9ea8ddb9f8e04893b56c10dbbb5af321 (DEFAULT_TABLE table=rebalance_test_table_11 [id=c46054f8169d4487b3c912c0681b1c3a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:25.830730 31645 tablet_service.cc:1467] Processing CreateTablet for tablet 9ea8ddb9f8e04893b56c10dbbb5af321 (DEFAULT_TABLE table=rebalance_test_table_11 [id=c46054f8169d4487b3c912c0681b1c3a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:25.831070 31907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ea8ddb9f8e04893b56c10dbbb5af321. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:25.831087 31645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ea8ddb9f8e04893b56c10dbbb5af321. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:25.832950 31775 tablet_service.cc:1467] Processing CreateTablet for tablet 9ea8ddb9f8e04893b56c10dbbb5af321 (DEFAULT_TABLE table=rebalance_test_table_11 [id=c46054f8169d4487b3c912c0681b1c3a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:25.833287 31775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ea8ddb9f8e04893b56c10dbbb5af321. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:25.833377 31906 tablet_service.cc:1467] Processing CreateTablet for tablet 88189733d9a14560971dbeb71543073f (DEFAULT_TABLE table=rebalance_test_table_11 [id=c46054f8169d4487b3c912c0681b1c3a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:25.833714 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88189733d9a14560971dbeb71543073f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:25.837015 31776 tablet_service.cc:1467] Processing CreateTablet for tablet 88189733d9a14560971dbeb71543073f (DEFAULT_TABLE table=rebalance_test_table_11 [id=c46054f8169d4487b3c912c0681b1c3a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:25.837332 31776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88189733d9a14560971dbeb71543073f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:25.837354 31516 tablet_service.cc:1467] Processing CreateTablet for tablet 88189733d9a14560971dbeb71543073f (DEFAULT_TABLE table=rebalance_test_table_11 [id=c46054f8169d4487b3c912c0681b1c3a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:25.837807 31516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88189733d9a14560971dbeb71543073f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:25.840765 31514 tablet_service.cc:1467] Processing CreateTablet for tablet 55fc18e4cc98446ca50c1dd2b6a9a89d (DEFAULT_TABLE table=rebalance_test_table_11 [id=c46054f8169d4487b3c912c0681b1c3a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:25.841078 31514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55fc18e4cc98446ca50c1dd2b6a9a89d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:25.847645 977 tablet_bootstrap.cc:492] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:25.849179 977 tablet_bootstrap.cc:654] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:25.851609 979 tablet_bootstrap.cc:492] T 88189733d9a14560971dbeb71543073f P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:25.853544 979 tablet_bootstrap.cc:654] T 88189733d9a14560971dbeb71543073f P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:25.853945 981 tablet_bootstrap.cc:492] T 88189733d9a14560971dbeb71543073f P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:25.855571 981 tablet_bootstrap.cc:654] T 88189733d9a14560971dbeb71543073f P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:25.855677 979 tablet_bootstrap.cc:492] T 88189733d9a14560971dbeb71543073f P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:25.855810 977 tablet_bootstrap.cc:492] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:25.855850 979 ts_tablet_manager.cc:1367] T 88189733d9a14560971dbeb71543073f P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:26:25.855973 977 ts_tablet_manager.cc:1367] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20240502 21:26:25.856797 978 tablet_bootstrap.cc:492] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:25.856863 977 raft_consensus.cc:348] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:25.857060 977 raft_consensus.cc:374] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:25.857121 977 raft_consensus.cc:729] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:25.856834 979 raft_consensus.cc:348] T 88189733d9a14560971dbeb71543073f P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:25.857342 979 raft_consensus.cc:374] T 88189733d9a14560971dbeb71543073f P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:25.857313 977 consensus_queue.cc:260] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:25.857492 979 raft_consensus.cc:729] T 88189733d9a14560971dbeb71543073f P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:25.857813 979 consensus_queue.cc:260] T 88189733d9a14560971dbeb71543073f P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:25.858403 977 ts_tablet_manager.cc:1398] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:25.858585 978 tablet_bootstrap.cc:654] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:25.858635 977 tablet_bootstrap.cc:492] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:25.859371 979 ts_tablet_manager.cc:1398] T 88189733d9a14560971dbeb71543073f P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:25.859694 979 tablet_bootstrap.cc:492] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:25.859936 977 tablet_bootstrap.cc:654] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:25.862107 979 tablet_bootstrap.cc:654] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:25.868573 981 tablet_bootstrap.cc:492] T 88189733d9a14560971dbeb71543073f P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:25.868662 977 tablet_bootstrap.cc:492] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:25.868743 981 ts_tablet_manager.cc:1367] T 88189733d9a14560971dbeb71543073f P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.001s
I20240502 21:26:25.868798 977 ts_tablet_manager.cc:1367] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20240502 21:26:25.869449 977 raft_consensus.cc:348] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:25.869491 981 raft_consensus.cc:348] T 88189733d9a14560971dbeb71543073f P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:25.869616 977 raft_consensus.cc:374] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:25.869681 977 raft_consensus.cc:729] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:25.869695 981 raft_consensus.cc:374] T 88189733d9a14560971dbeb71543073f P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:25.869783 981 raft_consensus.cc:729] T 88189733d9a14560971dbeb71543073f P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:25.869895 977 consensus_queue.cc:260] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:25.870016 979 tablet_bootstrap.cc:492] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:25.870167 979 ts_tablet_manager.cc:1367] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.002s
I20240502 21:26:25.870057 981 consensus_queue.cc:260] T 88189733d9a14560971dbeb71543073f P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:25.870455 977 ts_tablet_manager.cc:1398] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:25.870744 981 ts_tablet_manager.cc:1398] T 88189733d9a14560971dbeb71543073f P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:25.871013 981 tablet_bootstrap.cc:492] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:25.872565 981 tablet_bootstrap.cc:654] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:25.873896 978 tablet_bootstrap.cc:492] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:25.873769 979 raft_consensus.cc:348] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:25.874099 978 ts_tablet_manager.cc:1367] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.003s
I20240502 21:26:25.874116 979 raft_consensus.cc:374] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:25.874187 979 raft_consensus.cc:729] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:25.874629 979 consensus_queue.cc:260] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:25.874886 978 raft_consensus.cc:348] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:25.875128 978 raft_consensus.cc:374] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:25.875254 978 raft_consensus.cc:729] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:25.875526 978 consensus_queue.cc:260] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:25.876273 978 ts_tablet_manager.cc:1398] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:25.876683 978 tablet_bootstrap.cc:492] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:25.878595 978 tablet_bootstrap.cc:654] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:25.882169 979 ts_tablet_manager.cc:1398] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20240502 21:26:25.884359 981 tablet_bootstrap.cc:492] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:25.884518 981 ts_tablet_manager.cc:1367] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.008s
I20240502 21:26:25.885090 981 raft_consensus.cc:348] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:25.885304 981 raft_consensus.cc:374] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:25.885370 981 raft_consensus.cc:729] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:25.885541 981 consensus_queue.cc:260] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:25.885862 981 ts_tablet_manager.cc:1398] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:26:25.886191 981 tablet_bootstrap.cc:492] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:25.886678 978 tablet_bootstrap.cc:492] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:25.887312 981 tablet_bootstrap.cc:654] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:25.887612 978 ts_tablet_manager.cc:1367] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20240502 21:26:25.888298 978 raft_consensus.cc:348] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:25.888568 978 raft_consensus.cc:374] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:25.888664 978 raft_consensus.cc:729] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:25.888981 978 consensus_queue.cc:260] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:25.889607 978 ts_tablet_manager.cc:1398] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:25.892930 981 tablet_bootstrap.cc:492] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:25.893056 981 ts_tablet_manager.cc:1367] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20240502 21:26:25.893589 981 raft_consensus.cc:348] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:25.893713 981 raft_consensus.cc:374] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:25.893796 981 raft_consensus.cc:729] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:25.893988 981 consensus_queue.cc:260] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:25.894400 981 ts_tablet_manager.cc:1398] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:26:25.913071 896 raft_consensus.cc:482] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:25.913210 896 raft_consensus.cc:504] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:25.914026 896 leader_election.cc:288] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:25.914502 31667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea8ddb9f8e04893b56c10dbbb5af321" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:25.914556 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea8ddb9f8e04893b56c10dbbb5af321" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:25.914703 31667 raft_consensus.cc:2454] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:25.914736 31925 raft_consensus.cc:2454] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:25.915061 31733 leader_election.cc:302] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:25.915504 896 raft_consensus.cc:2781] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:25.915604 896 raft_consensus.cc:482] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:25.915678 896 raft_consensus.cc:3037] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:25.918304 896 raft_consensus.cc:504] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:25.918823 896 leader_election.cc:288] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Requested vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:25.919148 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea8ddb9f8e04893b56c10dbbb5af321" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:25.919306 31925 raft_consensus.cc:3037] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:25.919312 31667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ea8ddb9f8e04893b56c10dbbb5af321" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:25.919488 31667 raft_consensus.cc:3037] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:25.921165 31925 raft_consensus.cc:2454] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:25.921494 31733 leader_election.cc:302] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:25.921761 896 raft_consensus.cc:2781] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:25.921923 896 raft_consensus.cc:686] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Becoming Leader. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Running, Role: LEADER
I20240502 21:26:25.922268 31667 raft_consensus.cc:2454] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:25.922240 896 consensus_queue.cc:237] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:25.926290 31392 catalog_manager.cc:5484] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: term changed from 0 to 1, leader changed from <none> to 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131). New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:25.945932 885 raft_consensus.cc:482] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:25.946137 885 raft_consensus.cc:504] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:25.946705 885 leader_election.cc:288] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:25.947132 31667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fc18e4cc98446ca50c1dd2b6a9a89d" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:25.947135 31795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fc18e4cc98446ca50c1dd2b6a9a89d" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:25.947319 31795 raft_consensus.cc:2454] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:25.947320 31667 raft_consensus.cc:2454] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:25.947710 31472 leader_election.cc:302] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:25.947959 885 raft_consensus.cc:2781] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:25.948027 885 raft_consensus.cc:482] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:25.948132 885 raft_consensus.cc:3037] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:25.950048 885 raft_consensus.cc:504] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:25.950717 885 leader_election.cc:288] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Requested vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:25.951043 31795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fc18e4cc98446ca50c1dd2b6a9a89d" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:25.951112 31667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fc18e4cc98446ca50c1dd2b6a9a89d" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:25.951212 31795 raft_consensus.cc:3037] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:25.951236 31667 raft_consensus.cc:3037] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:25.953011 31667 raft_consensus.cc:2454] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:25.953091 31795 raft_consensus.cc:2454] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:25.953331 31472 leader_election.cc:302] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:25.953562 885 raft_consensus.cc:2781] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:25.953716 885 raft_consensus.cc:686] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:26:25.953965 885 consensus_queue.cc:237] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:25.956337 31392 catalog_manager.cc:5484] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 0 to 1, leader changed from <none> to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:26.219604 720 raft_consensus.cc:482] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:26.219749 720 raft_consensus.cc:504] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:26.220376 720 leader_election.cc:288] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:26.220715 31795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88189733d9a14560971dbeb71543073f" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:26.220798 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88189733d9a14560971dbeb71543073f" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:26.220960 31795 raft_consensus.cc:2454] T 88189733d9a14560971dbeb71543073f P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:26.221024 31536 raft_consensus.cc:2454] T 88189733d9a14560971dbeb71543073f P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:26.221307 31862 leader_election.cc:302] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:26.221625 720 raft_consensus.cc:2781] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:26.221735 720 raft_consensus.cc:482] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:26.221799 720 raft_consensus.cc:3037] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:26.223450 720 raft_consensus.cc:504] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:26.223881 720 leader_election.cc:288] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Requested vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:26.224257 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88189733d9a14560971dbeb71543073f" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:26:26.224256 31795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88189733d9a14560971dbeb71543073f" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:26.224454 31795 raft_consensus.cc:3037] T 88189733d9a14560971dbeb71543073f P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:26.224462 31536 raft_consensus.cc:3037] T 88189733d9a14560971dbeb71543073f P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:26.226032 31795 raft_consensus.cc:2454] T 88189733d9a14560971dbeb71543073f P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:26:26.226058 31536 raft_consensus.cc:2454] T 88189733d9a14560971dbeb71543073f P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:26:26.226387 31862 leader_election.cc:302] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:26.226629 720 raft_consensus.cc:2781] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:26.226935 720 raft_consensus.cc:686] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Running, Role: LEADER
I20240502 21:26:26.227151 720 consensus_queue.cc:237] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:26.229754 31392 catalog_manager.cc:5484] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132). New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:26.439234 896 consensus_queue.cc:1035] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:26.445685 885 consensus_queue.cc:1035] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:26.460148 896 consensus_queue.cc:1035] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:26.465065 885 consensus_queue.cc:1035] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:26.624717 995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:26.624980 995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:26.631435 995 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240502 21:26:26.673475 993 consensus_queue.cc:1035] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:26.679260 993 consensus_queue.cc:1035] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 272.323 ms
time per row: 0.136161 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240502 21:26:27.920632 31389 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalance_test_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:27.929523 31776 tablet_service.cc:1467] Processing CreateTablet for tablet dc734c0b3a094207856d5cd56c0e73b8 (DEFAULT_TABLE table=rebalance_test_table_12 [id=2ef1b694f2424f519337e0d5c334e4d5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:27.929976 31776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc734c0b3a094207856d5cd56c0e73b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:27.930205 31647 tablet_service.cc:1467] Processing CreateTablet for tablet dc734c0b3a094207856d5cd56c0e73b8 (DEFAULT_TABLE table=rebalance_test_table_12 [id=2ef1b694f2424f519337e0d5c334e4d5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:27.930346 31514 tablet_service.cc:1467] Processing CreateTablet for tablet cc8ef061e4224c8d94a06e47dd51a5d1 (DEFAULT_TABLE table=rebalance_test_table_12 [id=2ef1b694f2424f519337e0d5c334e4d5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:27.930573 31647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc734c0b3a094207856d5cd56c0e73b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:27.930689 31514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc8ef061e4224c8d94a06e47dd51a5d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:27.932967 31516 tablet_service.cc:1467] Processing CreateTablet for tablet dc734c0b3a094207856d5cd56c0e73b8 (DEFAULT_TABLE table=rebalance_test_table_12 [id=2ef1b694f2424f519337e0d5c334e4d5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:27.932955 31907 tablet_service.cc:1467] Processing CreateTablet for tablet cc8ef061e4224c8d94a06e47dd51a5d1 (DEFAULT_TABLE table=rebalance_test_table_12 [id=2ef1b694f2424f519337e0d5c334e4d5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:27.933326 31516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc734c0b3a094207856d5cd56c0e73b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:27.933333 31907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc8ef061e4224c8d94a06e47dd51a5d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:27.936794 31906 tablet_service.cc:1467] Processing CreateTablet for tablet bb64e0d78397429d959535f570658594 (DEFAULT_TABLE table=rebalance_test_table_12 [id=2ef1b694f2424f519337e0d5c334e4d5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:27.936772 31645 tablet_service.cc:1467] Processing CreateTablet for tablet cc8ef061e4224c8d94a06e47dd51a5d1 (DEFAULT_TABLE table=rebalance_test_table_12 [id=2ef1b694f2424f519337e0d5c334e4d5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:27.936930 31777 tablet_service.cc:1467] Processing CreateTablet for tablet bb64e0d78397429d959535f570658594 (DEFAULT_TABLE table=rebalance_test_table_12 [id=2ef1b694f2424f519337e0d5c334e4d5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:27.937130 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb64e0d78397429d959535f570658594. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:27.937182 31645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc8ef061e4224c8d94a06e47dd51a5d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:27.937230 31777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb64e0d78397429d959535f570658594. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:27.940804 31646 tablet_service.cc:1467] Processing CreateTablet for tablet bb64e0d78397429d959535f570658594 (DEFAULT_TABLE table=rebalance_test_table_12 [id=2ef1b694f2424f519337e0d5c334e4d5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:27.941133 31646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb64e0d78397429d959535f570658594. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:27.946352 1067 tablet_bootstrap.cc:492] T cc8ef061e4224c8d94a06e47dd51a5d1 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:27.948879 1067 tablet_bootstrap.cc:654] T cc8ef061e4224c8d94a06e47dd51a5d1 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:27.951766 1068 tablet_bootstrap.cc:492] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:27.953805 1068 tablet_bootstrap.cc:654] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:27.959064 1067 tablet_bootstrap.cc:492] T cc8ef061e4224c8d94a06e47dd51a5d1 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:27.959995 1067 ts_tablet_manager.cc:1367] T cc8ef061e4224c8d94a06e47dd51a5d1 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20240502 21:26:27.961159 1067 raft_consensus.cc:348] T cc8ef061e4224c8d94a06e47dd51a5d1 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:27.961591 1067 raft_consensus.cc:374] T cc8ef061e4224c8d94a06e47dd51a5d1 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:27.961808 1067 raft_consensus.cc:729] T cc8ef061e4224c8d94a06e47dd51a5d1 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:27.962224 1067 consensus_queue.cc:260] T cc8ef061e4224c8d94a06e47dd51a5d1 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:27.963336 1067 ts_tablet_manager.cc:1398] T cc8ef061e4224c8d94a06e47dd51a5d1 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:27.963769 1068 tablet_bootstrap.cc:492] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:27.963915 1067 tablet_bootstrap.cc:492] T dc734c0b3a094207856d5cd56c0e73b8 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:27.965799 1068 ts_tablet_manager.cc:1367] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.006s
I20240502 21:26:27.965901 1067 tablet_bootstrap.cc:654] T dc734c0b3a094207856d5cd56c0e73b8 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:27.967096 1070 tablet_bootstrap.cc:492] T dc734c0b3a094207856d5cd56c0e73b8 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:27.968675 1070 tablet_bootstrap.cc:654] T dc734c0b3a094207856d5cd56c0e73b8 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:27.970041 1068 raft_consensus.cc:348] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:27.970283 1068 raft_consensus.cc:374] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:27.970424 1068 raft_consensus.cc:729] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:27.970714 1068 consensus_queue.cc:260] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:27.972196 1068 ts_tablet_manager.cc:1398] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 21:26:27.972540 1068 tablet_bootstrap.cc:492] T cc8ef061e4224c8d94a06e47dd51a5d1 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:27.974006 1068 tablet_bootstrap.cc:654] T cc8ef061e4224c8d94a06e47dd51a5d1 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:27.974828 1066 tablet_bootstrap.cc:492] T dc734c0b3a094207856d5cd56c0e73b8 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:27.977249 1067 tablet_bootstrap.cc:492] T dc734c0b3a094207856d5cd56c0e73b8 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:27.977524 1067 ts_tablet_manager.cc:1367] T dc734c0b3a094207856d5cd56c0e73b8 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20240502 21:26:27.978171 1067 raft_consensus.cc:348] T dc734c0b3a094207856d5cd56c0e73b8 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:27.978339 1067 raft_consensus.cc:374] T dc734c0b3a094207856d5cd56c0e73b8 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:27.978518 1067 raft_consensus.cc:729] T dc734c0b3a094207856d5cd56c0e73b8 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:27.978796 1067 consensus_queue.cc:260] T dc734c0b3a094207856d5cd56c0e73b8 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:27.979388 1067 ts_tablet_manager.cc:1398] T dc734c0b3a094207856d5cd56c0e73b8 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:27.980155 1066 tablet_bootstrap.cc:654] T dc734c0b3a094207856d5cd56c0e73b8 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:27.982270 1070 tablet_bootstrap.cc:492] T dc734c0b3a094207856d5cd56c0e73b8 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:27.982465 1070 ts_tablet_manager.cc:1367] T dc734c0b3a094207856d5cd56c0e73b8 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240502 21:26:27.983424 1070 raft_consensus.cc:348] T dc734c0b3a094207856d5cd56c0e73b8 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:27.983697 1070 raft_consensus.cc:374] T dc734c0b3a094207856d5cd56c0e73b8 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:27.983829 1070 raft_consensus.cc:729] T dc734c0b3a094207856d5cd56c0e73b8 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:27.984107 1070 consensus_queue.cc:260] T dc734c0b3a094207856d5cd56c0e73b8 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:27.984822 1070 ts_tablet_manager.cc:1398] T dc734c0b3a094207856d5cd56c0e73b8 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:27.988704 1070 tablet_bootstrap.cc:492] T bb64e0d78397429d959535f570658594 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:27.990386 1070 tablet_bootstrap.cc:654] T bb64e0d78397429d959535f570658594 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:28.001507 1068 tablet_bootstrap.cc:492] T cc8ef061e4224c8d94a06e47dd51a5d1 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:28.001672 1068 ts_tablet_manager.cc:1367] T cc8ef061e4224c8d94a06e47dd51a5d1 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.005s
I20240502 21:26:28.002005 1066 tablet_bootstrap.cc:492] T dc734c0b3a094207856d5cd56c0e73b8 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:28.002167 1066 ts_tablet_manager.cc:1367] T dc734c0b3a094207856d5cd56c0e73b8 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.018s
I20240502 21:26:28.002358 1068 raft_consensus.cc:348] T cc8ef061e4224c8d94a06e47dd51a5d1 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:28.002534 1068 raft_consensus.cc:374] T cc8ef061e4224c8d94a06e47dd51a5d1 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:28.002612 1068 raft_consensus.cc:729] T cc8ef061e4224c8d94a06e47dd51a5d1 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:28.002848 1068 consensus_queue.cc:260] T cc8ef061e4224c8d94a06e47dd51a5d1 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:28.002842 1066 raft_consensus.cc:348] T dc734c0b3a094207856d5cd56c0e73b8 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:28.003023 1066 raft_consensus.cc:374] T dc734c0b3a094207856d5cd56c0e73b8 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:28.003754 1068 ts_tablet_manager.cc:1398] T cc8ef061e4224c8d94a06e47dd51a5d1 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:28.003922 1070 tablet_bootstrap.cc:492] T bb64e0d78397429d959535f570658594 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:28.004078 1070 ts_tablet_manager.cc:1367] T bb64e0d78397429d959535f570658594 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20240502 21:26:28.004884 1066 raft_consensus.cc:729] T dc734c0b3a094207856d5cd56c0e73b8 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:28.004701 1070 raft_consensus.cc:348] T bb64e0d78397429d959535f570658594 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:28.005133 1070 raft_consensus.cc:374] T bb64e0d78397429d959535f570658594 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:28.005198 1070 raft_consensus.cc:729] T bb64e0d78397429d959535f570658594 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:28.005384 1070 consensus_queue.cc:260] T bb64e0d78397429d959535f570658594 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:28.005822 1070 ts_tablet_manager.cc:1398] T bb64e0d78397429d959535f570658594 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:28.006084 1066 consensus_queue.cc:260] T dc734c0b3a094207856d5cd56c0e73b8 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:28.007365 1066 ts_tablet_manager.cc:1398] T dc734c0b3a094207856d5cd56c0e73b8 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 21:26:28.007651 1066 tablet_bootstrap.cc:492] T cc8ef061e4224c8d94a06e47dd51a5d1 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:28.009903 1066 tablet_bootstrap.cc:654] T cc8ef061e4224c8d94a06e47dd51a5d1 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:28.011843 1066 tablet_bootstrap.cc:492] T cc8ef061e4224c8d94a06e47dd51a5d1 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:28.012041 1066 ts_tablet_manager.cc:1367] T cc8ef061e4224c8d94a06e47dd51a5d1 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:26:28.012499 1000 raft_consensus.cc:482] T dc734c0b3a094207856d5cd56c0e73b8 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:28.012638 1000 raft_consensus.cc:504] T dc734c0b3a094207856d5cd56c0e73b8 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:28.012753 1066 raft_consensus.cc:348] T cc8ef061e4224c8d94a06e47dd51a5d1 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:28.012955 1066 raft_consensus.cc:374] T cc8ef061e4224c8d94a06e47dd51a5d1 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:28.013036 1066 raft_consensus.cc:729] T cc8ef061e4224c8d94a06e47dd51a5d1 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:28.013298 1000 leader_election.cc:288] T dc734c0b3a094207856d5cd56c0e73b8 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:28.013312 1066 consensus_queue.cc:260] T cc8ef061e4224c8d94a06e47dd51a5d1 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:28.013764 1066 ts_tablet_manager.cc:1398] T cc8ef061e4224c8d94a06e47dd51a5d1 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:28.013692 31795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc734c0b3a094207856d5cd56c0e73b8" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:28.013897 31795 raft_consensus.cc:2454] T dc734c0b3a094207856d5cd56c0e73b8 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:28.013965 1066 tablet_bootstrap.cc:492] T bb64e0d78397429d959535f570658594 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:28.014348 31471 leader_election.cc:302] T dc734c0b3a094207856d5cd56c0e73b8 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:28.014402 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc734c0b3a094207856d5cd56c0e73b8" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:28.014591 31666 raft_consensus.cc:2454] T dc734c0b3a094207856d5cd56c0e73b8 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:28.015259 1066 tablet_bootstrap.cc:654] T bb64e0d78397429d959535f570658594 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:28.016261 1000 raft_consensus.cc:2781] T dc734c0b3a094207856d5cd56c0e73b8 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:28.016371 1000 raft_consensus.cc:482] T dc734c0b3a094207856d5cd56c0e73b8 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:28.016475 1000 raft_consensus.cc:3037] T dc734c0b3a094207856d5cd56c0e73b8 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:28.016886 1066 tablet_bootstrap.cc:492] T bb64e0d78397429d959535f570658594 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:28.017014 1066 ts_tablet_manager.cc:1367] T bb64e0d78397429d959535f570658594 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:28.017530 1066 raft_consensus.cc:348] T bb64e0d78397429d959535f570658594 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:28.017663 1066 raft_consensus.cc:374] T bb64e0d78397429d959535f570658594 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:28.017798 1066 raft_consensus.cc:729] T bb64e0d78397429d959535f570658594 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:28.018023 1066 consensus_queue.cc:260] T bb64e0d78397429d959535f570658594 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:28.018494 1066 ts_tablet_manager.cc:1398] T bb64e0d78397429d959535f570658594 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 21:26:28.019066 1000 raft_consensus.cc:504] T dc734c0b3a094207856d5cd56c0e73b8 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:28.020149 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc734c0b3a094207856d5cd56c0e73b8" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:28.020179 31795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc734c0b3a094207856d5cd56c0e73b8" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:28.020350 31666 raft_consensus.cc:3037] T dc734c0b3a094207856d5cd56c0e73b8 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:28.020354 31795 raft_consensus.cc:3037] T dc734c0b3a094207856d5cd56c0e73b8 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:28.022989 31666 raft_consensus.cc:2454] T dc734c0b3a094207856d5cd56c0e73b8 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:28.023012 31795 raft_consensus.cc:2454] T dc734c0b3a094207856d5cd56c0e73b8 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:28.023428 31472 leader_election.cc:302] T dc734c0b3a094207856d5cd56c0e73b8 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:28.023567 1000 leader_election.cc:288] T dc734c0b3a094207856d5cd56c0e73b8 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Requested vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:28.023746 1000 raft_consensus.cc:2781] T dc734c0b3a094207856d5cd56c0e73b8 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:28.023896 1000 raft_consensus.cc:686] T dc734c0b3a094207856d5cd56c0e73b8 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:26:28.024142 1000 consensus_queue.cc:237] T dc734c0b3a094207856d5cd56c0e73b8 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:28.026692 31389 catalog_manager.cc:5484] T dc734c0b3a094207856d5cd56c0e73b8 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 0 to 1, leader changed from <none> to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:28.158860 1082 raft_consensus.cc:482] T cc8ef061e4224c8d94a06e47dd51a5d1 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:28.159010 1082 raft_consensus.cc:504] T cc8ef061e4224c8d94a06e47dd51a5d1 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:28.159747 1082 leader_election.cc:288] T cc8ef061e4224c8d94a06e47dd51a5d1 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:28.160115 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc8ef061e4224c8d94a06e47dd51a5d1" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:28.160317 31926 raft_consensus.cc:2454] T cc8ef061e4224c8d94a06e47dd51a5d1 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:26:28.160252 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc8ef061e4224c8d94a06e47dd51a5d1" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:28.160502 31536 raft_consensus.cc:2454] T cc8ef061e4224c8d94a06e47dd51a5d1 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:26:28.160786 31603 leader_election.cc:302] T cc8ef061e4224c8d94a06e47dd51a5d1 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:28.161054 1082 raft_consensus.cc:2781] T cc8ef061e4224c8d94a06e47dd51a5d1 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:28.161164 1082 raft_consensus.cc:482] T cc8ef061e4224c8d94a06e47dd51a5d1 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:28.161255 1082 raft_consensus.cc:3037] T cc8ef061e4224c8d94a06e47dd51a5d1 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:28.163327 1082 raft_consensus.cc:504] T cc8ef061e4224c8d94a06e47dd51a5d1 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:28.163825 1082 leader_election.cc:288] T cc8ef061e4224c8d94a06e47dd51a5d1 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Requested vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:28.164165 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc8ef061e4224c8d94a06e47dd51a5d1" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:28.164191 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc8ef061e4224c8d94a06e47dd51a5d1" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:26:28.164320 31926 raft_consensus.cc:3037] T cc8ef061e4224c8d94a06e47dd51a5d1 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:28.164357 31536 raft_consensus.cc:3037] T cc8ef061e4224c8d94a06e47dd51a5d1 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:28.166242 31926 raft_consensus.cc:2454] T cc8ef061e4224c8d94a06e47dd51a5d1 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:28.166429 31536 raft_consensus.cc:2454] T cc8ef061e4224c8d94a06e47dd51a5d1 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:28.166589 31603 leader_election.cc:302] T cc8ef061e4224c8d94a06e47dd51a5d1 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:28.166873 1082 raft_consensus.cc:2781] T cc8ef061e4224c8d94a06e47dd51a5d1 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:28.167024 1082 raft_consensus.cc:686] T cc8ef061e4224c8d94a06e47dd51a5d1 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Becoming Leader. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Running, Role: LEADER
I20240502 21:26:28.167344 1082 consensus_queue.cc:237] T cc8ef061e4224c8d94a06e47dd51a5d1 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:28.170452 31389 catalog_manager.cc:5484] T cc8ef061e4224c8d94a06e47dd51a5d1 P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130). New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:28.288971 993 raft_consensus.cc:482] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:28.289103 993 raft_consensus.cc:504] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:28.289598 993 leader_election.cc:288] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:28.289961 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb64e0d78397429d959535f570658594" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:28.289945 31795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb64e0d78397429d959535f570658594" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:28.290149 31665 raft_consensus.cc:2454] T bb64e0d78397429d959535f570658594 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:28.290192 31795 raft_consensus.cc:2454] T bb64e0d78397429d959535f570658594 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:28.290551 31863 leader_election.cc:302] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 08d7a0680d244b55b93d17f9ab57b84c, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:28.290823 993 raft_consensus.cc:2781] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:28.290930 993 raft_consensus.cc:482] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:28.291010 993 raft_consensus.cc:3037] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:28.292928 993 raft_consensus.cc:504] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:28.293460 993 leader_election.cc:288] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Requested vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:28.293807 31795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb64e0d78397429d959535f570658594" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:28.294005 31795 raft_consensus.cc:3037] T bb64e0d78397429d959535f570658594 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:28.294211 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb64e0d78397429d959535f570658594" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:28.294415 31665 raft_consensus.cc:3037] T bb64e0d78397429d959535f570658594 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:28.295922 31795 raft_consensus.cc:2454] T bb64e0d78397429d959535f570658594 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:26:28.296360 31862 leader_election.cc:302] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:28.296506 31665 raft_consensus.cc:2454] T bb64e0d78397429d959535f570658594 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:26:28.296686 993 raft_consensus.cc:2781] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:28.296859 993 raft_consensus.cc:686] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Running, Role: LEADER
I20240502 21:26:28.297107 993 consensus_queue.cc:237] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:28.299986 31389 catalog_manager.cc:5484] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132). New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:28.437906 1000 consensus_queue.cc:1035] T dc734c0b3a094207856d5cd56c0e73b8 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:28.444923 1000 consensus_queue.cc:1035] T dc734c0b3a094207856d5cd56c0e73b8 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:28.565824 1082 consensus_queue.cc:1035] T cc8ef061e4224c8d94a06e47dd51a5d1 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:28.571954 1082 consensus_queue.cc:1035] T cc8ef061e4224c8d94a06e47dd51a5d1 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:28.629678 1086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:28.629941 1086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:28.636870 1086 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_12'
I20240502 21:26:28.808130 993 consensus_queue.cc:1035] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:28.859969 1133 consensus_queue.cc:1035] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 249.958 ms
time per row: 0.124979 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 21:26:29.910701 31392 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalance_test_table_13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:29.921716 31516 tablet_service.cc:1467] Processing CreateTablet for tablet 5b1dcae186f542cea92c9191fa8ef951 (DEFAULT_TABLE table=rebalance_test_table_13 [id=c7511c25586c4ba086fcfffa353da437]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:29.922222 31516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b1dcae186f542cea92c9191fa8ef951. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:29.922519 31906 tablet_service.cc:1467] Processing CreateTablet for tablet 5b1dcae186f542cea92c9191fa8ef951 (DEFAULT_TABLE table=rebalance_test_table_13 [id=c7511c25586c4ba086fcfffa353da437]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:29.922914 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b1dcae186f542cea92c9191fa8ef951. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:29.922801 31776 tablet_service.cc:1467] Processing CreateTablet for tablet 5b1dcae186f542cea92c9191fa8ef951 (DEFAULT_TABLE table=rebalance_test_table_13 [id=c7511c25586c4ba086fcfffa353da437]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:29.923169 31776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b1dcae186f542cea92c9191fa8ef951. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:29.927565 31775 tablet_service.cc:1467] Processing CreateTablet for tablet c5c344f8f84f4a73b853b4f908b93964 (DEFAULT_TABLE table=rebalance_test_table_13 [id=c7511c25586c4ba086fcfffa353da437]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:29.927960 31775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5c344f8f84f4a73b853b4f908b93964. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:29.928269 31777 tablet_service.cc:1467] Processing CreateTablet for tablet 759b7529d7d84baab1a4622d34f803bf (DEFAULT_TABLE table=rebalance_test_table_13 [id=c7511c25586c4ba086fcfffa353da437]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:29.928647 31777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 759b7529d7d84baab1a4622d34f803bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:29.931082 1155 tablet_bootstrap.cc:492] T 5b1dcae186f542cea92c9191fa8ef951 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:29.932807 1155 tablet_bootstrap.cc:654] T 5b1dcae186f542cea92c9191fa8ef951 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:29.933720 31647 tablet_service.cc:1467] Processing CreateTablet for tablet c5c344f8f84f4a73b853b4f908b93964 (DEFAULT_TABLE table=rebalance_test_table_13 [id=c7511c25586c4ba086fcfffa353da437]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:29.934140 31647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5c344f8f84f4a73b853b4f908b93964. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:29.934402 31514 tablet_service.cc:1467] Processing CreateTablet for tablet 759b7529d7d84baab1a4622d34f803bf (DEFAULT_TABLE table=rebalance_test_table_13 [id=c7511c25586c4ba086fcfffa353da437]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:29.934900 31514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 759b7529d7d84baab1a4622d34f803bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:29.937017 31515 tablet_service.cc:1467] Processing CreateTablet for tablet c5c344f8f84f4a73b853b4f908b93964 (DEFAULT_TABLE table=rebalance_test_table_13 [id=c7511c25586c4ba086fcfffa353da437]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:29.937472 31515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5c344f8f84f4a73b853b4f908b93964. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:29.942108 31906 tablet_service.cc:1467] Processing CreateTablet for tablet 759b7529d7d84baab1a4622d34f803bf (DEFAULT_TABLE table=rebalance_test_table_13 [id=c7511c25586c4ba086fcfffa353da437]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:29.942622 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 759b7529d7d84baab1a4622d34f803bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:29.955263 1157 tablet_bootstrap.cc:492] T 759b7529d7d84baab1a4622d34f803bf P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:29.958873 1159 tablet_bootstrap.cc:492] T c5c344f8f84f4a73b853b4f908b93964 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:29.959724 1157 tablet_bootstrap.cc:654] T 759b7529d7d84baab1a4622d34f803bf P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:29.962522 1158 tablet_bootstrap.cc:492] T 5b1dcae186f542cea92c9191fa8ef951 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:29.963652 1159 tablet_bootstrap.cc:654] T c5c344f8f84f4a73b853b4f908b93964 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:29.964738 1155 tablet_bootstrap.cc:492] T 5b1dcae186f542cea92c9191fa8ef951 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:29.964879 1155 ts_tablet_manager.cc:1367] T 5b1dcae186f542cea92c9191fa8ef951 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.034s user 0.013s sys 0.004s
I20240502 21:26:29.965646 1155 raft_consensus.cc:348] T 5b1dcae186f542cea92c9191fa8ef951 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:29.965844 1155 raft_consensus.cc:374] T 5b1dcae186f542cea92c9191fa8ef951 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:29.965940 1155 raft_consensus.cc:729] T 5b1dcae186f542cea92c9191fa8ef951 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:29.966311 1158 tablet_bootstrap.cc:654] T 5b1dcae186f542cea92c9191fa8ef951 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:29.966181 1155 consensus_queue.cc:260] T 5b1dcae186f542cea92c9191fa8ef951 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:29.967821 1155 ts_tablet_manager.cc:1398] T 5b1dcae186f542cea92c9191fa8ef951 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:26:29.968089 1155 tablet_bootstrap.cc:492] T 759b7529d7d84baab1a4622d34f803bf P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:29.968982 1157 tablet_bootstrap.cc:492] T 759b7529d7d84baab1a4622d34f803bf P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:29.969210 1157 ts_tablet_manager.cc:1367] T 759b7529d7d84baab1a4622d34f803bf P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.009s
I20240502 21:26:29.970013 1157 raft_consensus.cc:348] T 759b7529d7d84baab1a4622d34f803bf P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:29.970273 1157 raft_consensus.cc:374] T 759b7529d7d84baab1a4622d34f803bf P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:29.970417 1157 raft_consensus.cc:729] T 759b7529d7d84baab1a4622d34f803bf P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:29.970476 1158 tablet_bootstrap.cc:492] T 5b1dcae186f542cea92c9191fa8ef951 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:29.970733 1157 consensus_queue.cc:260] T 759b7529d7d84baab1a4622d34f803bf P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:29.970888 1155 tablet_bootstrap.cc:654] T 759b7529d7d84baab1a4622d34f803bf P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:29.971455 1157 ts_tablet_manager.cc:1398] T 759b7529d7d84baab1a4622d34f803bf P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:29.971774 1157 tablet_bootstrap.cc:492] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:29.973421 1157 tablet_bootstrap.cc:654] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:29.974570 1159 tablet_bootstrap.cc:492] T c5c344f8f84f4a73b853b4f908b93964 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:29.974557 1158 ts_tablet_manager.cc:1367] T 5b1dcae186f542cea92c9191fa8ef951 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240502 21:26:29.974769 1159 ts_tablet_manager.cc:1367] T c5c344f8f84f4a73b853b4f908b93964 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.003s
I20240502 21:26:29.975625 1159 raft_consensus.cc:348] T c5c344f8f84f4a73b853b4f908b93964 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:29.975924 1159 raft_consensus.cc:374] T c5c344f8f84f4a73b853b4f908b93964 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:29.975919 1158 raft_consensus.cc:348] T 5b1dcae186f542cea92c9191fa8ef951 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:29.976169 1158 raft_consensus.cc:374] T 5b1dcae186f542cea92c9191fa8ef951 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:29.976054 1159 raft_consensus.cc:729] T c5c344f8f84f4a73b853b4f908b93964 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:29.976361 1158 raft_consensus.cc:729] T 5b1dcae186f542cea92c9191fa8ef951 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:29.976678 1158 consensus_queue.cc:260] T 5b1dcae186f542cea92c9191fa8ef951 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:29.977393 1158 ts_tablet_manager.cc:1398] T 5b1dcae186f542cea92c9191fa8ef951 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:26:29.976667 1159 consensus_queue.cc:260] T c5c344f8f84f4a73b853b4f908b93964 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:29.977744 1158 tablet_bootstrap.cc:492] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:29.978080 1159 ts_tablet_manager.cc:1398] T c5c344f8f84f4a73b853b4f908b93964 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:26:29.979935 1158 tablet_bootstrap.cc:654] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:29.980466 1157 tablet_bootstrap.cc:492] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:29.980691 1157 ts_tablet_manager.cc:1367] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20240502 21:26:29.981532 1157 raft_consensus.cc:348] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:29.981973 1157 raft_consensus.cc:374] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:29.982123 1157 raft_consensus.cc:729] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:29.982491 1157 consensus_queue.cc:260] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:29.983202 1157 ts_tablet_manager.cc:1398] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:29.983465 1157 tablet_bootstrap.cc:492] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:29.984946 1157 tablet_bootstrap.cc:654] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:29.996578 1158 tablet_bootstrap.cc:492] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:29.997376 1149 raft_consensus.cc:482] T 759b7529d7d84baab1a4622d34f803bf P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:29.997525 1149 raft_consensus.cc:504] T 759b7529d7d84baab1a4622d34f803bf P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:29.998308 1149 leader_election.cc:288] T 759b7529d7d84baab1a4622d34f803bf P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:29.998458 1155 tablet_bootstrap.cc:492] T 759b7529d7d84baab1a4622d34f803bf P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:29.998602 1155 ts_tablet_manager.cc:1367] T 759b7529d7d84baab1a4622d34f803bf P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.003s
I20240502 21:26:29.998989 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "759b7529d7d84baab1a4622d34f803bf" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:29.999344 1155 raft_consensus.cc:348] T 759b7529d7d84baab1a4622d34f803bf P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:29.999567 1155 raft_consensus.cc:374] T 759b7529d7d84baab1a4622d34f803bf P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:29.999648 1155 raft_consensus.cc:729] T 759b7529d7d84baab1a4622d34f803bf P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:29.999872 1155 consensus_queue.cc:260] T 759b7529d7d84baab1a4622d34f803bf P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
W20240502 21:26:29.999974 31733 leader_election.cc:341] T 759b7529d7d84baab1a4622d34f803bf P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:29.999974 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "759b7529d7d84baab1a4622d34f803bf" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:30.000748 1158 ts_tablet_manager.cc:1367] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.012s
W20240502 21:26:30.000854 31733 leader_election.cc:341] T 759b7529d7d84baab1a4622d34f803bf P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:30.000981 31733 leader_election.cc:302] T 759b7529d7d84baab1a4622d34f803bf P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c; no voters: 2d50f33557944bb480ea32199d9ee4f1, a44f41b0b6644fb6aa29476aed8c964f
I20240502 21:26:30.001224 1149 raft_consensus.cc:2726] T 759b7529d7d84baab1a4622d34f803bf P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:26:30.001753 1155 ts_tablet_manager.cc:1398] T 759b7529d7d84baab1a4622d34f803bf P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240502 21:26:30.001788 1158 raft_consensus.cc:348] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:30.001988 1158 raft_consensus.cc:374] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:30.002099 1158 raft_consensus.cc:729] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:30.002272 1157 tablet_bootstrap.cc:492] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:30.002481 1157 ts_tablet_manager.cc:1367] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.011s
I20240502 21:26:30.002354 1158 consensus_queue.cc:260] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:30.003147 1158 ts_tablet_manager.cc:1398] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:30.003458 1158 tablet_bootstrap.cc:492] T c5c344f8f84f4a73b853b4f908b93964 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:30.003345 1157 raft_consensus.cc:348] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:30.004251 1157 raft_consensus.cc:374] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:30.004400 1157 raft_consensus.cc:729] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:30.005797 1157 consensus_queue.cc:260] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:30.006506 1157 ts_tablet_manager.cc:1398] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 21:26:30.007130 1158 tablet_bootstrap.cc:654] T c5c344f8f84f4a73b853b4f908b93964 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:30.008806 1158 tablet_bootstrap.cc:492] T c5c344f8f84f4a73b853b4f908b93964 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:30.008971 1158 ts_tablet_manager.cc:1367] T c5c344f8f84f4a73b853b4f908b93964 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20240502 21:26:30.009542 1158 raft_consensus.cc:348] T c5c344f8f84f4a73b853b4f908b93964 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:30.009683 1158 raft_consensus.cc:374] T c5c344f8f84f4a73b853b4f908b93964 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:30.009770 1158 raft_consensus.cc:729] T c5c344f8f84f4a73b853b4f908b93964 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:30.009968 1158 consensus_queue.cc:260] T c5c344f8f84f4a73b853b4f908b93964 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:30.010529 1158 ts_tablet_manager.cc:1398] T c5c344f8f84f4a73b853b4f908b93964 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 21:26:30.022200 1149 raft_consensus.cc:482] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:30.022399 1149 raft_consensus.cc:504] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:30.023010 1149 leader_election.cc:288] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:30.023276 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1dcae186f542cea92c9191fa8ef951" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:30.023377 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1dcae186f542cea92c9191fa8ef951" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:30.023507 31925 raft_consensus.cc:2454] T 5b1dcae186f542cea92c9191fa8ef951 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:30.023603 31535 raft_consensus.cc:2454] T 5b1dcae186f542cea92c9191fa8ef951 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:30.023856 31733 leader_election.cc:302] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:30.024089 1149 raft_consensus.cc:2781] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:30.024227 1149 raft_consensus.cc:482] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:30.024317 1149 raft_consensus.cc:3037] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:30.026283 1149 raft_consensus.cc:504] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:30.026715 1149 leader_election.cc:288] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Requested vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:30.027158 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1dcae186f542cea92c9191fa8ef951" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:30.027244 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1dcae186f542cea92c9191fa8ef951" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:26:30.027318 31925 raft_consensus.cc:3037] T 5b1dcae186f542cea92c9191fa8ef951 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:30.027428 31535 raft_consensus.cc:3037] T 5b1dcae186f542cea92c9191fa8ef951 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:30.029482 31925 raft_consensus.cc:2454] T 5b1dcae186f542cea92c9191fa8ef951 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:30.029690 31535 raft_consensus.cc:2454] T 5b1dcae186f542cea92c9191fa8ef951 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:30.029901 31733 leader_election.cc:302] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:30.030227 1149 raft_consensus.cc:2781] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:30.030362 1149 raft_consensus.cc:686] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Becoming Leader. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Running, Role: LEADER
I20240502 21:26:30.031140 1149 consensus_queue.cc:237] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:30.034189 31392 catalog_manager.cc:5484] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: term changed from 0 to 1, leader changed from <none> to 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131). New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:30.149492 1149 raft_consensus.cc:482] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:30.149660 1149 raft_consensus.cc:504] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:30.150357 1149 leader_election.cc:288] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:30.150652 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c344f8f84f4a73b853b4f908b93964" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:30.150849 31665 raft_consensus.cc:2454] T c5c344f8f84f4a73b853b4f908b93964 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:30.150777 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c344f8f84f4a73b853b4f908b93964" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:30.151010 31535 raft_consensus.cc:2454] T c5c344f8f84f4a73b853b4f908b93964 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:30.151233 31733 leader_election.cc:302] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 08d7a0680d244b55b93d17f9ab57b84c; no voters:
I20240502 21:26:30.151479 1149 raft_consensus.cc:2781] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:30.151587 1149 raft_consensus.cc:482] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:30.151710 1149 raft_consensus.cc:3037] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:30.154076 1149 raft_consensus.cc:504] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:30.154784 1149 leader_election.cc:288] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Requested vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:30.155144 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c344f8f84f4a73b853b4f908b93964" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:30.155263 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c344f8f84f4a73b853b4f908b93964" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:26:30.155386 31665 raft_consensus.cc:3037] T c5c344f8f84f4a73b853b4f908b93964 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:30.155473 31535 raft_consensus.cc:3037] T c5c344f8f84f4a73b853b4f908b93964 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:30.158156 31535 raft_consensus.cc:2454] T c5c344f8f84f4a73b853b4f908b93964 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:30.158154 31665 raft_consensus.cc:2454] T c5c344f8f84f4a73b853b4f908b93964 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:30.158497 31733 leader_election.cc:302] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 08d7a0680d244b55b93d17f9ab57b84c; no voters:
I20240502 21:26:30.158761 1149 raft_consensus.cc:2781] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:30.158917 1149 raft_consensus.cc:686] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Becoming Leader. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Running, Role: LEADER
I20240502 21:26:30.159152 1149 consensus_queue.cc:237] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:30.162987 31392 catalog_manager.cc:5484] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: term changed from 0 to 1, leader changed from <none> to 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131). New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:30.244333 1080 raft_consensus.cc:482] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:30.244534 1080 raft_consensus.cc:504] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:30.245112 1080 leader_election.cc:288] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:30.245536 31795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "759b7529d7d84baab1a4622d34f803bf" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:30.245743 31795 raft_consensus.cc:2454] T 759b7529d7d84baab1a4622d34f803bf P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:30.245798 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "759b7529d7d84baab1a4622d34f803bf" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:30.246037 31926 raft_consensus.cc:2454] T 759b7529d7d84baab1a4622d34f803bf P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:30.246358 31471 leader_election.cc:302] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:30.246587 1080 raft_consensus.cc:2781] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:30.246685 1080 raft_consensus.cc:482] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:30.246764 1080 raft_consensus.cc:3037] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:30.248505 1080 raft_consensus.cc:504] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:30.249083 1080 leader_election.cc:288] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Requested vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:30.249486 31795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "759b7529d7d84baab1a4622d34f803bf" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:30.249653 31795 raft_consensus.cc:3037] T 759b7529d7d84baab1a4622d34f803bf P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:30.251327 31795 raft_consensus.cc:2454] T 759b7529d7d84baab1a4622d34f803bf P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:30.251722 31471 leader_election.cc:302] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:30.252040 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "759b7529d7d84baab1a4622d34f803bf" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:30.252213 31926 raft_consensus.cc:3037] T 759b7529d7d84baab1a4622d34f803bf P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:30.252316 1080 raft_consensus.cc:2781] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:30.252481 1080 raft_consensus.cc:686] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:26:30.252784 1080 consensus_queue.cc:237] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:30.255018 31926 raft_consensus.cc:2454] T 759b7529d7d84baab1a4622d34f803bf P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:30.263123 31392 catalog_manager.cc:5484] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 0 to 1, leader changed from <none> to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:30.454726 1061 consensus_queue.cc:1035] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:30.462566 1149 consensus_queue.cc:1035] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:30.563233 1061 consensus_queue.cc:1035] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:30.573879 1149 consensus_queue.cc:1035] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:30.615803 1173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:30.616058 1173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:30.623893 1173 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:26:30.638015 1080 consensus_queue.cc:1035] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:30.642829 1000 consensus_queue.cc:1035] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_13'
INSERT report
rows total: 2000
time total: 213.662 ms
time per row: 0.106831 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240502 21:26:31.867931 31393 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalance_test_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:31.876317 31647 tablet_service.cc:1467] Processing CreateTablet for tablet 1e9b7eae127146a18a27d26d103b04d3 (DEFAULT_TABLE table=rebalance_test_table_14 [id=c880d77c61494966b13524216c035e27]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:31.876574 31516 tablet_service.cc:1467] Processing CreateTablet for tablet 1e9b7eae127146a18a27d26d103b04d3 (DEFAULT_TABLE table=rebalance_test_table_14 [id=c880d77c61494966b13524216c035e27]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:31.876796 31647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e9b7eae127146a18a27d26d103b04d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:31.876775 31906 tablet_service.cc:1467] Processing CreateTablet for tablet 1e9b7eae127146a18a27d26d103b04d3 (DEFAULT_TABLE table=rebalance_test_table_14 [id=c880d77c61494966b13524216c035e27]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:31.876940 31516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e9b7eae127146a18a27d26d103b04d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:31.877121 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e9b7eae127146a18a27d26d103b04d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:31.881083 31777 tablet_service.cc:1467] Processing CreateTablet for tablet e69d22e2c0b348b1bc439c1618bd59f6 (DEFAULT_TABLE table=rebalance_test_table_14 [id=c880d77c61494966b13524216c035e27]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:31.881436 31777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e69d22e2c0b348b1bc439c1618bd59f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:31.882783 1238 tablet_bootstrap.cc:492] T 1e9b7eae127146a18a27d26d103b04d3 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:31.882990 31646 tablet_service.cc:1467] Processing CreateTablet for tablet e69d22e2c0b348b1bc439c1618bd59f6 (DEFAULT_TABLE table=rebalance_test_table_14 [id=c880d77c61494966b13524216c035e27]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:31.883368 31646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e69d22e2c0b348b1bc439c1618bd59f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:31.883646 31907 tablet_service.cc:1467] Processing CreateTablet for tablet e69d22e2c0b348b1bc439c1618bd59f6 (DEFAULT_TABLE table=rebalance_test_table_14 [id=c880d77c61494966b13524216c035e27]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:31.883975 31907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e69d22e2c0b348b1bc439c1618bd59f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:31.884620 1238 tablet_bootstrap.cc:654] T 1e9b7eae127146a18a27d26d103b04d3 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:31.888015 31905 tablet_service.cc:1467] Processing CreateTablet for tablet d3b744acac964cbd82abc3306f016baf (DEFAULT_TABLE table=rebalance_test_table_14 [id=c880d77c61494966b13524216c035e27]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:31.888334 31905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3b744acac964cbd82abc3306f016baf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:31.893965 31646 tablet_service.cc:1467] Processing CreateTablet for tablet d3b744acac964cbd82abc3306f016baf (DEFAULT_TABLE table=rebalance_test_table_14 [id=c880d77c61494966b13524216c035e27]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:31.894340 31646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3b744acac964cbd82abc3306f016baf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:31.894675 31515 tablet_service.cc:1467] Processing CreateTablet for tablet d3b744acac964cbd82abc3306f016baf (DEFAULT_TABLE table=rebalance_test_table_14 [id=c880d77c61494966b13524216c035e27]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:31.895108 31515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3b744acac964cbd82abc3306f016baf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:31.900506 1238 tablet_bootstrap.cc:492] T 1e9b7eae127146a18a27d26d103b04d3 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:31.900636 1238 ts_tablet_manager.cc:1367] T 1e9b7eae127146a18a27d26d103b04d3 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.012s
I20240502 21:26:31.901309 1238 raft_consensus.cc:348] T 1e9b7eae127146a18a27d26d103b04d3 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:31.901505 1238 raft_consensus.cc:374] T 1e9b7eae127146a18a27d26d103b04d3 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:31.901638 1238 raft_consensus.cc:729] T 1e9b7eae127146a18a27d26d103b04d3 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:31.901669 1241 tablet_bootstrap.cc:492] T 1e9b7eae127146a18a27d26d103b04d3 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:31.901880 1238 consensus_queue.cc:260] T 1e9b7eae127146a18a27d26d103b04d3 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:31.902519 1238 ts_tablet_manager.cc:1398] T 1e9b7eae127146a18a27d26d103b04d3 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:31.903080 1240 tablet_bootstrap.cc:492] T e69d22e2c0b348b1bc439c1618bd59f6 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:31.903460 1238 tablet_bootstrap.cc:492] T d3b744acac964cbd82abc3306f016baf P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:31.904590 1238 tablet_bootstrap.cc:654] T d3b744acac964cbd82abc3306f016baf P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:31.904872 1240 tablet_bootstrap.cc:654] T e69d22e2c0b348b1bc439c1618bd59f6 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:31.906677 1240 tablet_bootstrap.cc:492] T e69d22e2c0b348b1bc439c1618bd59f6 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:31.906847 1240 ts_tablet_manager.cc:1367] T e69d22e2c0b348b1bc439c1618bd59f6 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:26:31.907316 1238 tablet_bootstrap.cc:492] T d3b744acac964cbd82abc3306f016baf P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:31.907742 1238 ts_tablet_manager.cc:1367] T d3b744acac964cbd82abc3306f016baf P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 21:26:31.907666 1240 raft_consensus.cc:348] T e69d22e2c0b348b1bc439c1618bd59f6 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:31.907975 1240 raft_consensus.cc:374] T e69d22e2c0b348b1bc439c1618bd59f6 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:31.908133 1240 raft_consensus.cc:729] T e69d22e2c0b348b1bc439c1618bd59f6 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:31.908406 1240 consensus_queue.cc:260] T e69d22e2c0b348b1bc439c1618bd59f6 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:31.908480 1238 raft_consensus.cc:348] T d3b744acac964cbd82abc3306f016baf P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:31.908659 1238 raft_consensus.cc:374] T d3b744acac964cbd82abc3306f016baf P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:31.908743 1238 raft_consensus.cc:729] T d3b744acac964cbd82abc3306f016baf P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:31.909128 1238 consensus_queue.cc:260] T d3b744acac964cbd82abc3306f016baf P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:31.909864 1238 ts_tablet_manager.cc:1398] T d3b744acac964cbd82abc3306f016baf P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:31.910079 1240 ts_tablet_manager.cc:1398] T e69d22e2c0b348b1bc439c1618bd59f6 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:31.910176 1238 tablet_bootstrap.cc:492] T e69d22e2c0b348b1bc439c1618bd59f6 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:31.911656 1238 tablet_bootstrap.cc:654] T e69d22e2c0b348b1bc439c1618bd59f6 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:31.912065 1241 tablet_bootstrap.cc:654] T 1e9b7eae127146a18a27d26d103b04d3 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:31.913391 1238 tablet_bootstrap.cc:492] T e69d22e2c0b348b1bc439c1618bd59f6 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:31.913504 1238 ts_tablet_manager.cc:1367] T e69d22e2c0b348b1bc439c1618bd59f6 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:26:31.913607 1242 tablet_bootstrap.cc:492] T 1e9b7eae127146a18a27d26d103b04d3 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:31.914068 1238 raft_consensus.cc:348] T e69d22e2c0b348b1bc439c1618bd59f6 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:31.914232 1238 raft_consensus.cc:374] T e69d22e2c0b348b1bc439c1618bd59f6 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:31.914301 1238 raft_consensus.cc:729] T e69d22e2c0b348b1bc439c1618bd59f6 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:31.914474 1238 consensus_queue.cc:260] T e69d22e2c0b348b1bc439c1618bd59f6 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:31.915087 1238 ts_tablet_manager.cc:1398] T e69d22e2c0b348b1bc439c1618bd59f6 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240502 21:26:31.915141 1242 tablet_bootstrap.cc:654] T 1e9b7eae127146a18a27d26d103b04d3 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:31.916666 1242 tablet_bootstrap.cc:492] T 1e9b7eae127146a18a27d26d103b04d3 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:31.917726 1242 ts_tablet_manager.cc:1367] T 1e9b7eae127146a18a27d26d103b04d3 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:26:31.918403 1242 raft_consensus.cc:348] T 1e9b7eae127146a18a27d26d103b04d3 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:31.918622 1242 raft_consensus.cc:374] T 1e9b7eae127146a18a27d26d103b04d3 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:31.918726 1242 raft_consensus.cc:729] T 1e9b7eae127146a18a27d26d103b04d3 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:31.918920 1242 consensus_queue.cc:260] T 1e9b7eae127146a18a27d26d103b04d3 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:31.919677 1242 ts_tablet_manager.cc:1398] T 1e9b7eae127146a18a27d26d103b04d3 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:31.919930 1242 tablet_bootstrap.cc:492] T d3b744acac964cbd82abc3306f016baf P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:31.921258 1242 tablet_bootstrap.cc:654] T d3b744acac964cbd82abc3306f016baf P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:31.922984 1241 tablet_bootstrap.cc:492] T 1e9b7eae127146a18a27d26d103b04d3 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:31.923133 1242 tablet_bootstrap.cc:492] T d3b744acac964cbd82abc3306f016baf P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:31.923185 1241 ts_tablet_manager.cc:1367] T 1e9b7eae127146a18a27d26d103b04d3 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20240502 21:26:31.923390 1242 ts_tablet_manager.cc:1367] T d3b744acac964cbd82abc3306f016baf P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 21:26:31.924005 1241 raft_consensus.cc:348] T 1e9b7eae127146a18a27d26d103b04d3 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:31.924180 1241 raft_consensus.cc:374] T 1e9b7eae127146a18a27d26d103b04d3 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:31.924127 1242 raft_consensus.cc:348] T d3b744acac964cbd82abc3306f016baf P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:31.924276 1241 raft_consensus.cc:729] T 1e9b7eae127146a18a27d26d103b04d3 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:31.924373 1242 raft_consensus.cc:374] T d3b744acac964cbd82abc3306f016baf P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:31.924460 1242 raft_consensus.cc:729] T d3b744acac964cbd82abc3306f016baf P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:31.924500 1241 consensus_queue.cc:260] T 1e9b7eae127146a18a27d26d103b04d3 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:31.924705 1242 consensus_queue.cc:260] T d3b744acac964cbd82abc3306f016baf P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:31.925184 1242 ts_tablet_manager.cc:1398] T d3b744acac964cbd82abc3306f016baf P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:26:31.932868 1241 ts_tablet_manager.cc:1398] T 1e9b7eae127146a18a27d26d103b04d3 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240502 21:26:31.933290 1241 tablet_bootstrap.cc:492] T e69d22e2c0b348b1bc439c1618bd59f6 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:31.934569 1241 tablet_bootstrap.cc:654] T e69d22e2c0b348b1bc439c1618bd59f6 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:31.935911 1241 tablet_bootstrap.cc:492] T e69d22e2c0b348b1bc439c1618bd59f6 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:31.936043 1241 ts_tablet_manager.cc:1367] T e69d22e2c0b348b1bc439c1618bd59f6 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:31.936589 1241 raft_consensus.cc:348] T e69d22e2c0b348b1bc439c1618bd59f6 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:31.936771 1241 raft_consensus.cc:374] T e69d22e2c0b348b1bc439c1618bd59f6 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:31.936849 1241 raft_consensus.cc:729] T e69d22e2c0b348b1bc439c1618bd59f6 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:31.937037 1241 consensus_queue.cc:260] T e69d22e2c0b348b1bc439c1618bd59f6 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:31.937399 1241 ts_tablet_manager.cc:1398] T e69d22e2c0b348b1bc439c1618bd59f6 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 21:26:31.937587 1241 tablet_bootstrap.cc:492] T d3b744acac964cbd82abc3306f016baf P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:31.938699 1241 tablet_bootstrap.cc:654] T d3b744acac964cbd82abc3306f016baf P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:31.940555 1241 tablet_bootstrap.cc:492] T d3b744acac964cbd82abc3306f016baf P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:31.940696 1241 ts_tablet_manager.cc:1367] T d3b744acac964cbd82abc3306f016baf P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:31.941130 1241 raft_consensus.cc:348] T d3b744acac964cbd82abc3306f016baf P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:31.941251 1241 raft_consensus.cc:374] T d3b744acac964cbd82abc3306f016baf P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:31.941334 1241 raft_consensus.cc:729] T d3b744acac964cbd82abc3306f016baf P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:31.941493 1241 consensus_queue.cc:260] T d3b744acac964cbd82abc3306f016baf P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:31.941896 1241 ts_tablet_manager.cc:1398] T d3b744acac964cbd82abc3306f016baf P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:26:31.998080 1061 raft_consensus.cc:482] T e69d22e2c0b348b1bc439c1618bd59f6 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:31.998225 1061 raft_consensus.cc:504] T e69d22e2c0b348b1bc439c1618bd59f6 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:31.998783 1061 leader_election.cc:288] T e69d22e2c0b348b1bc439c1618bd59f6 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:31.999126 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e69d22e2c0b348b1bc439c1618bd59f6" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:31.999218 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e69d22e2c0b348b1bc439c1618bd59f6" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:31.999297 31666 raft_consensus.cc:2454] T e69d22e2c0b348b1bc439c1618bd59f6 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:31.999399 31926 raft_consensus.cc:2454] T e69d22e2c0b348b1bc439c1618bd59f6 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:31.999658 31733 leader_election.cc:302] T e69d22e2c0b348b1bc439c1618bd59f6 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 08d7a0680d244b55b93d17f9ab57b84c; no voters:
I20240502 21:26:31.999894 1061 raft_consensus.cc:2781] T e69d22e2c0b348b1bc439c1618bd59f6 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:31.999990 1061 raft_consensus.cc:482] T e69d22e2c0b348b1bc439c1618bd59f6 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:32.000068 1061 raft_consensus.cc:3037] T e69d22e2c0b348b1bc439c1618bd59f6 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:32.001775 1061 raft_consensus.cc:504] T e69d22e2c0b348b1bc439c1618bd59f6 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:32.002226 1061 leader_election.cc:288] T e69d22e2c0b348b1bc439c1618bd59f6 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Requested vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:32.002593 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e69d22e2c0b348b1bc439c1618bd59f6" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:32.002751 31666 raft_consensus.cc:3037] T e69d22e2c0b348b1bc439c1618bd59f6 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:32.002676 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e69d22e2c0b348b1bc439c1618bd59f6" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:32.002857 31926 raft_consensus.cc:3037] T e69d22e2c0b348b1bc439c1618bd59f6 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:32.004997 31666 raft_consensus.cc:2454] T e69d22e2c0b348b1bc439c1618bd59f6 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:32.005038 31926 raft_consensus.cc:2454] T e69d22e2c0b348b1bc439c1618bd59f6 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:32.005342 31733 leader_election.cc:302] T e69d22e2c0b348b1bc439c1618bd59f6 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 08d7a0680d244b55b93d17f9ab57b84c; no voters:
I20240502 21:26:32.005746 1061 raft_consensus.cc:2781] T e69d22e2c0b348b1bc439c1618bd59f6 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:32.005870 1061 raft_consensus.cc:686] T e69d22e2c0b348b1bc439c1618bd59f6 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Becoming Leader. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Running, Role: LEADER
I20240502 21:26:32.006261 1061 consensus_queue.cc:237] T e69d22e2c0b348b1bc439c1618bd59f6 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:32.008824 31390 catalog_manager.cc:5484] T e69d22e2c0b348b1bc439c1618bd59f6 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: term changed from 0 to 1, leader changed from <none> to 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131). New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:32.108212 1132 raft_consensus.cc:482] T d3b744acac964cbd82abc3306f016baf P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:32.108341 1132 raft_consensus.cc:504] T d3b744acac964cbd82abc3306f016baf P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:32.109074 1132 leader_election.cc:288] T d3b744acac964cbd82abc3306f016baf P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:32.109357 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b744acac964cbd82abc3306f016baf" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:32.109488 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b744acac964cbd82abc3306f016baf" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:32.109577 31666 raft_consensus.cc:2454] T d3b744acac964cbd82abc3306f016baf P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:32.109714 31535 raft_consensus.cc:2454] T d3b744acac964cbd82abc3306f016baf P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:32.109963 31863 leader_election.cc:302] T d3b744acac964cbd82abc3306f016baf P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:32.110173 1132 raft_consensus.cc:2781] T d3b744acac964cbd82abc3306f016baf P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:32.110275 1132 raft_consensus.cc:482] T d3b744acac964cbd82abc3306f016baf P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:32.110363 1132 raft_consensus.cc:3037] T d3b744acac964cbd82abc3306f016baf P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:32.112543 1132 raft_consensus.cc:504] T d3b744acac964cbd82abc3306f016baf P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:32.113022 1132 leader_election.cc:288] T d3b744acac964cbd82abc3306f016baf P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Requested vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:32.113363 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b744acac964cbd82abc3306f016baf" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:32.113569 31666 raft_consensus.cc:3037] T d3b744acac964cbd82abc3306f016baf P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:32.113533 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b744acac964cbd82abc3306f016baf" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:26:32.113696 31535 raft_consensus.cc:3037] T d3b744acac964cbd82abc3306f016baf P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:32.115995 31666 raft_consensus.cc:2454] T d3b744acac964cbd82abc3306f016baf P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:26:32.116137 31535 raft_consensus.cc:2454] T d3b744acac964cbd82abc3306f016baf P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:26:32.116356 31863 leader_election.cc:302] T d3b744acac964cbd82abc3306f016baf P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:32.116609 1132 raft_consensus.cc:2781] T d3b744acac964cbd82abc3306f016baf P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:32.116941 1132 raft_consensus.cc:686] T d3b744acac964cbd82abc3306f016baf P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Running, Role: LEADER
I20240502 21:26:32.117228 1132 consensus_queue.cc:237] T d3b744acac964cbd82abc3306f016baf P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:32.119808 31390 catalog_manager.cc:5484] T d3b744acac964cbd82abc3306f016baf P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132). New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:32.172729 1000 raft_consensus.cc:482] T 1e9b7eae127146a18a27d26d103b04d3 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:32.172881 1000 raft_consensus.cc:504] T 1e9b7eae127146a18a27d26d103b04d3 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:32.173380 1000 leader_election.cc:288] T 1e9b7eae127146a18a27d26d103b04d3 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:32.173743 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e9b7eae127146a18a27d26d103b04d3" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:32.173787 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e9b7eae127146a18a27d26d103b04d3" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:32.173910 31666 raft_consensus.cc:2454] T 1e9b7eae127146a18a27d26d103b04d3 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:32.173993 31926 raft_consensus.cc:2454] T 1e9b7eae127146a18a27d26d103b04d3 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:32.174237 31472 leader_election.cc:302] T 1e9b7eae127146a18a27d26d103b04d3 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:32.174463 1000 raft_consensus.cc:2781] T 1e9b7eae127146a18a27d26d103b04d3 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:32.174572 1000 raft_consensus.cc:482] T 1e9b7eae127146a18a27d26d103b04d3 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:32.174655 1000 raft_consensus.cc:3037] T 1e9b7eae127146a18a27d26d103b04d3 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:32.176280 1000 raft_consensus.cc:504] T 1e9b7eae127146a18a27d26d103b04d3 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:32.176801 1000 leader_election.cc:288] T 1e9b7eae127146a18a27d26d103b04d3 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Requested vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:32.177192 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e9b7eae127146a18a27d26d103b04d3" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:32.177199 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e9b7eae127146a18a27d26d103b04d3" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:32.177348 31666 raft_consensus.cc:3037] T 1e9b7eae127146a18a27d26d103b04d3 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:32.177356 31926 raft_consensus.cc:3037] T 1e9b7eae127146a18a27d26d103b04d3 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:32.178906 31926 raft_consensus.cc:2454] T 1e9b7eae127146a18a27d26d103b04d3 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:32.178969 31666 raft_consensus.cc:2454] T 1e9b7eae127146a18a27d26d103b04d3 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:32.179270 31472 leader_election.cc:302] T 1e9b7eae127146a18a27d26d103b04d3 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:32.179540 1000 raft_consensus.cc:2781] T 1e9b7eae127146a18a27d26d103b04d3 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:32.179672 1000 raft_consensus.cc:686] T 1e9b7eae127146a18a27d26d103b04d3 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:26:32.179921 1000 consensus_queue.cc:237] T 1e9b7eae127146a18a27d26d103b04d3 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:32.182796 31390 catalog_manager.cc:5484] T 1e9b7eae127146a18a27d26d103b04d3 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 0 to 1, leader changed from <none> to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:32.414407 1061 consensus_queue.cc:1035] T e69d22e2c0b348b1bc439c1618bd59f6 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:32.420804 1252 consensus_queue.cc:1035] T e69d22e2c0b348b1bc439c1618bd59f6 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:32.510016 1257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:32.510370 1257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:32.521041 1257 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_14'
I20240502 21:26:32.573303 1132 consensus_queue.cc:1035] T d3b744acac964cbd82abc3306f016baf P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:32.583566 1000 consensus_queue.cc:1035] T 1e9b7eae127146a18a27d26d103b04d3 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:32.640267 1254 consensus_queue.cc:1035] T d3b744acac964cbd82abc3306f016baf P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:32.654498 1000 consensus_queue.cc:1035] T 1e9b7eae127146a18a27d26d103b04d3 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 292.218 ms
time per row: 0.146109 ms
I20240502 21:26:32.952889 31392 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalance_test_table_15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:32.960873 31777 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab82de23912483ea178376a5ba5cdc1 (DEFAULT_TABLE table=rebalance_test_table_15 [id=7ff12e5a4c674a5fb7e07cb593390fd1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:32.961321 31777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab82de23912483ea178376a5ba5cdc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:32.961341 31776 tablet_service.cc:1467] Processing CreateTablet for tablet 0b8d119ee8474a40ac08df61df5b8423 (DEFAULT_TABLE table=rebalance_test_table_15 [id=7ff12e5a4c674a5fb7e07cb593390fd1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:32.961648 31776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b8d119ee8474a40ac08df61df5b8423. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:32.963109 31646 tablet_service.cc:1467] Processing CreateTablet for tablet 0b8d119ee8474a40ac08df61df5b8423 (DEFAULT_TABLE table=rebalance_test_table_15 [id=7ff12e5a4c674a5fb7e07cb593390fd1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:32.963527 31646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b8d119ee8474a40ac08df61df5b8423. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:32.963490 31906 tablet_service.cc:1467] Processing CreateTablet for tablet 0b8d119ee8474a40ac08df61df5b8423 (DEFAULT_TABLE table=rebalance_test_table_15 [id=7ff12e5a4c674a5fb7e07cb593390fd1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:32.963851 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b8d119ee8474a40ac08df61df5b8423. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:32.965055 31775 tablet_service.cc:1467] Processing CreateTablet for tablet 83904952a5a541ecac508449e23e70dd (DEFAULT_TABLE table=rebalance_test_table_15 [id=7ff12e5a4c674a5fb7e07cb593390fd1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:32.965411 31775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83904952a5a541ecac508449e23e70dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:32.968746 31905 tablet_service.cc:1467] Processing CreateTablet for tablet 83904952a5a541ecac508449e23e70dd (DEFAULT_TABLE table=rebalance_test_table_15 [id=7ff12e5a4c674a5fb7e07cb593390fd1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:32.969122 31905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83904952a5a541ecac508449e23e70dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:32.972190 31647 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab82de23912483ea178376a5ba5cdc1 (DEFAULT_TABLE table=rebalance_test_table_15 [id=7ff12e5a4c674a5fb7e07cb593390fd1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:32.972697 31647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab82de23912483ea178376a5ba5cdc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:32.973692 31516 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab82de23912483ea178376a5ba5cdc1 (DEFAULT_TABLE table=rebalance_test_table_15 [id=7ff12e5a4c674a5fb7e07cb593390fd1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:32.974059 31516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab82de23912483ea178376a5ba5cdc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:32.975322 31515 tablet_service.cc:1467] Processing CreateTablet for tablet 83904952a5a541ecac508449e23e70dd (DEFAULT_TABLE table=rebalance_test_table_15 [id=7ff12e5a4c674a5fb7e07cb593390fd1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:32.975664 31515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83904952a5a541ecac508449e23e70dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:32.989008 1320 tablet_bootstrap.cc:492] T 0b8d119ee8474a40ac08df61df5b8423 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:32.989442 1322 tablet_bootstrap.cc:492] T 0b8d119ee8474a40ac08df61df5b8423 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:32.990617 1320 tablet_bootstrap.cc:654] T 0b8d119ee8474a40ac08df61df5b8423 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:32.991853 1322 tablet_bootstrap.cc:654] T 0b8d119ee8474a40ac08df61df5b8423 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:32.996822 1321 tablet_bootstrap.cc:492] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:32.998265 1321 tablet_bootstrap.cc:654] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:33.000041 1321 tablet_bootstrap.cc:492] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:33.000382 1321 ts_tablet_manager.cc:1367] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 21:26:33.001839 1321 raft_consensus.cc:348] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:33.002012 1321 raft_consensus.cc:374] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:33.002871 1321 raft_consensus.cc:729] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:33.003428 1321 consensus_queue.cc:260] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:33.004307 1321 ts_tablet_manager.cc:1398] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:26:33.004999 1321 tablet_bootstrap.cc:492] T 8ab82de23912483ea178376a5ba5cdc1 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:33.007764 1322 tablet_bootstrap.cc:492] T 0b8d119ee8474a40ac08df61df5b8423 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:33.007942 1322 ts_tablet_manager.cc:1367] T 0b8d119ee8474a40ac08df61df5b8423 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.004s
I20240502 21:26:33.008116 1323 tablet_bootstrap.cc:492] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:33.009230 1322 raft_consensus.cc:348] T 0b8d119ee8474a40ac08df61df5b8423 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:33.009513 1322 raft_consensus.cc:374] T 0b8d119ee8474a40ac08df61df5b8423 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:33.009774 1322 raft_consensus.cc:729] T 0b8d119ee8474a40ac08df61df5b8423 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:33.010044 1322 consensus_queue.cc:260] T 0b8d119ee8474a40ac08df61df5b8423 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:33.010538 1323 tablet_bootstrap.cc:654] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:33.010883 1322 ts_tablet_manager.cc:1398] T 0b8d119ee8474a40ac08df61df5b8423 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:26:33.011200 1322 tablet_bootstrap.cc:492] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:33.013034 1320 tablet_bootstrap.cc:492] T 0b8d119ee8474a40ac08df61df5b8423 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:33.013242 1320 ts_tablet_manager.cc:1367] T 0b8d119ee8474a40ac08df61df5b8423 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.014s
I20240502 21:26:33.013903 1320 raft_consensus.cc:348] T 0b8d119ee8474a40ac08df61df5b8423 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:33.014163 1320 raft_consensus.cc:374] T 0b8d119ee8474a40ac08df61df5b8423 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:33.014279 1320 raft_consensus.cc:729] T 0b8d119ee8474a40ac08df61df5b8423 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:33.014497 1320 consensus_queue.cc:260] T 0b8d119ee8474a40ac08df61df5b8423 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:33.015096 1320 ts_tablet_manager.cc:1398] T 0b8d119ee8474a40ac08df61df5b8423 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:26:33.015108 1322 tablet_bootstrap.cc:654] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:33.015463 1320 tablet_bootstrap.cc:492] T 83904952a5a541ecac508449e23e70dd P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:33.015882 1321 tablet_bootstrap.cc:654] T 8ab82de23912483ea178376a5ba5cdc1 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:33.016824 1320 tablet_bootstrap.cc:654] T 83904952a5a541ecac508449e23e70dd P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:33.019023 1321 tablet_bootstrap.cc:492] T 8ab82de23912483ea178376a5ba5cdc1 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:33.019194 1321 ts_tablet_manager.cc:1367] T 8ab82de23912483ea178376a5ba5cdc1 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.002s
I20240502 21:26:33.019896 1321 raft_consensus.cc:348] T 8ab82de23912483ea178376a5ba5cdc1 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:33.020116 1321 raft_consensus.cc:374] T 8ab82de23912483ea178376a5ba5cdc1 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:33.020200 1321 raft_consensus.cc:729] T 8ab82de23912483ea178376a5ba5cdc1 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:33.020498 1321 consensus_queue.cc:260] T 8ab82de23912483ea178376a5ba5cdc1 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:33.021651 1321 ts_tablet_manager.cc:1398] T 8ab82de23912483ea178376a5ba5cdc1 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:33.022617 1322 tablet_bootstrap.cc:492] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:33.022797 1322 ts_tablet_manager.cc:1367] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s
I20240502 21:26:33.023543 1322 raft_consensus.cc:348] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:33.023744 1322 raft_consensus.cc:374] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:33.023824 1322 raft_consensus.cc:729] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:33.024071 1322 consensus_queue.cc:260] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:33.024739 1322 ts_tablet_manager.cc:1398] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:33.025377 1320 tablet_bootstrap.cc:492] T 83904952a5a541ecac508449e23e70dd P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:33.025568 1320 ts_tablet_manager.cc:1367] T 83904952a5a541ecac508449e23e70dd P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20240502 21:26:33.025606 1323 tablet_bootstrap.cc:492] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:33.025738 1323 ts_tablet_manager.cc:1367] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.006s
I20240502 21:26:33.026378 1320 raft_consensus.cc:348] T 83904952a5a541ecac508449e23e70dd P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:33.026515 1323 raft_consensus.cc:348] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:33.026631 1320 raft_consensus.cc:374] T 83904952a5a541ecac508449e23e70dd P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:33.026700 1320 raft_consensus.cc:729] T 83904952a5a541ecac508449e23e70dd P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:33.026754 1323 raft_consensus.cc:374] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:33.026880 1323 raft_consensus.cc:729] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:33.026955 1320 consensus_queue.cc:260] T 83904952a5a541ecac508449e23e70dd P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:33.027540 1320 ts_tablet_manager.cc:1398] T 83904952a5a541ecac508449e23e70dd P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:33.027931 1320 tablet_bootstrap.cc:492] T 8ab82de23912483ea178376a5ba5cdc1 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:33.027127 1323 consensus_queue.cc:260] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:33.028714 1323 ts_tablet_manager.cc:1398] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:26:33.029083 1323 tablet_bootstrap.cc:492] T 83904952a5a541ecac508449e23e70dd P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:33.029435 1320 tablet_bootstrap.cc:654] T 8ab82de23912483ea178376a5ba5cdc1 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:33.030678 1323 tablet_bootstrap.cc:654] T 83904952a5a541ecac508449e23e70dd P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:33.037691 1323 tablet_bootstrap.cc:492] T 83904952a5a541ecac508449e23e70dd P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:33.037844 1323 ts_tablet_manager.cc:1367] T 83904952a5a541ecac508449e23e70dd P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20240502 21:26:33.038110 1320 tablet_bootstrap.cc:492] T 8ab82de23912483ea178376a5ba5cdc1 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:33.038233 1320 ts_tablet_manager.cc:1367] T 8ab82de23912483ea178376a5ba5cdc1 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20240502 21:26:33.038636 1323 raft_consensus.cc:348] T 83904952a5a541ecac508449e23e70dd P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:33.038851 1323 raft_consensus.cc:374] T 83904952a5a541ecac508449e23e70dd P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:33.038937 1323 raft_consensus.cc:729] T 83904952a5a541ecac508449e23e70dd P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:33.038870 1320 raft_consensus.cc:348] T 8ab82de23912483ea178376a5ba5cdc1 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:33.039054 1320 raft_consensus.cc:374] T 8ab82de23912483ea178376a5ba5cdc1 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:33.039160 1320 raft_consensus.cc:729] T 8ab82de23912483ea178376a5ba5cdc1 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:33.039180 1323 consensus_queue.cc:260] T 83904952a5a541ecac508449e23e70dd P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:33.039350 1320 consensus_queue.cc:260] T 8ab82de23912483ea178376a5ba5cdc1 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:33.039659 1323 ts_tablet_manager.cc:1398] T 83904952a5a541ecac508449e23e70dd P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:26:33.039820 1320 ts_tablet_manager.cc:1398] T 8ab82de23912483ea178376a5ba5cdc1 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:26:33.064574 1287 raft_consensus.cc:482] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:33.064697 1287 raft_consensus.cc:504] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:33.065316 1287 leader_election.cc:288] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:33.065719 31795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab82de23912483ea178376a5ba5cdc1" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:33.065743 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab82de23912483ea178376a5ba5cdc1" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:33.065922 31665 raft_consensus.cc:2454] T 8ab82de23912483ea178376a5ba5cdc1 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:33.065941 31795 raft_consensus.cc:2454] T 8ab82de23912483ea178376a5ba5cdc1 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:33.066345 31472 leader_election.cc:302] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:33.066654 1287 raft_consensus.cc:2781] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:33.066766 1287 raft_consensus.cc:482] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:33.066865 1287 raft_consensus.cc:3037] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:33.068753 1287 raft_consensus.cc:504] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:33.069383 1287 leader_election.cc:288] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Requested vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:33.069752 31795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab82de23912483ea178376a5ba5cdc1" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:33.069802 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab82de23912483ea178376a5ba5cdc1" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:33.069929 31795 raft_consensus.cc:3037] T 8ab82de23912483ea178376a5ba5cdc1 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:33.069975 31665 raft_consensus.cc:3037] T 8ab82de23912483ea178376a5ba5cdc1 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:33.071976 31795 raft_consensus.cc:2454] T 8ab82de23912483ea178376a5ba5cdc1 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:33.071980 31665 raft_consensus.cc:2454] T 8ab82de23912483ea178376a5ba5cdc1 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:33.072494 31471 leader_election.cc:302] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:33.072821 1287 raft_consensus.cc:2781] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:33.072964 1287 raft_consensus.cc:686] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:26:33.073205 1287 consensus_queue.cc:237] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:33.076097 31394 catalog_manager.cc:5484] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 0 to 1, leader changed from <none> to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:33.154507 1132 raft_consensus.cc:482] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:33.154676 1132 raft_consensus.cc:504] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:33.155437 1132 leader_election.cc:288] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:33.155686 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83904952a5a541ecac508449e23e70dd" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:33.155797 31795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83904952a5a541ecac508449e23e70dd" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:33.155890 31536 raft_consensus.cc:2454] T 83904952a5a541ecac508449e23e70dd P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:33.155998 31795 raft_consensus.cc:2454] T 83904952a5a541ecac508449e23e70dd P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:33.156294 31863 leader_election.cc:302] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:33.156563 1132 raft_consensus.cc:2781] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:33.156659 1132 raft_consensus.cc:482] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:33.156737 1132 raft_consensus.cc:3037] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:33.158581 1132 raft_consensus.cc:504] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:33.159041 1132 leader_election.cc:288] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Requested vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:33.159248 1334 raft_consensus.cc:482] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:33.159443 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83904952a5a541ecac508449e23e70dd" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:26:33.159477 31795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83904952a5a541ecac508449e23e70dd" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:33.159449 1334 raft_consensus.cc:504] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:33.159632 31795 raft_consensus.cc:3037] T 83904952a5a541ecac508449e23e70dd P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:33.159627 31536 raft_consensus.cc:3037] T 83904952a5a541ecac508449e23e70dd P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:33.160459 1334 leader_election.cc:288] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:33.160933 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b8d119ee8474a40ac08df61df5b8423" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:33.161168 31797 raft_consensus.cc:2454] T 0b8d119ee8474a40ac08df61df5b8423 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:26:33.161159 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b8d119ee8474a40ac08df61df5b8423" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:33.161398 31926 raft_consensus.cc:2454] T 0b8d119ee8474a40ac08df61df5b8423 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:26:33.161761 31603 leader_election.cc:302] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:33.162060 1334 raft_consensus.cc:2781] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:33.162189 1334 raft_consensus.cc:482] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:33.162165 31536 raft_consensus.cc:2454] T 83904952a5a541ecac508449e23e70dd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:26:33.162273 1334 raft_consensus.cc:3037] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:33.162590 31795 raft_consensus.cc:2454] T 83904952a5a541ecac508449e23e70dd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:26:33.162946 31862 leader_election.cc:302] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:33.163208 1132 raft_consensus.cc:2781] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:33.163369 1132 raft_consensus.cc:686] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Running, Role: LEADER
I20240502 21:26:33.163636 1132 consensus_queue.cc:237] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:33.164803 1334 raft_consensus.cc:504] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:33.165344 1334 leader_election.cc:288] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Requested vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:33.165599 31795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b8d119ee8474a40ac08df61df5b8423" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:33.165764 31795 raft_consensus.cc:3037] T 0b8d119ee8474a40ac08df61df5b8423 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:33.165731 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b8d119ee8474a40ac08df61df5b8423" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:33.165896 31926 raft_consensus.cc:3037] T 0b8d119ee8474a40ac08df61df5b8423 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:33.167021 31394 catalog_manager.cc:5484] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132). New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:33.168108 31926 raft_consensus.cc:2454] T 0b8d119ee8474a40ac08df61df5b8423 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:33.168114 31795 raft_consensus.cc:2454] T 0b8d119ee8474a40ac08df61df5b8423 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:33.168510 31602 leader_election.cc:302] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 08d7a0680d244b55b93d17f9ab57b84c, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:33.168864 1334 raft_consensus.cc:2781] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:33.168999 1334 raft_consensus.cc:686] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Becoming Leader. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Running, Role: LEADER
I20240502 21:26:33.169231 1334 consensus_queue.cc:237] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:33.172123 31394 catalog_manager.cc:5484] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130). New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } }
W20240502 21:26:33.518245 1337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:33.518605 1337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:33.529687 1337 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:26:33.533820 1000 consensus_queue.cc:1035] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:33.553144 1000 consensus_queue.cc:1035] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_15'
I20240502 21:26:33.597658 1334 consensus_queue.cc:1035] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:33.601830 1230 consensus_queue.cc:1035] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:33.709872 31795 raft_consensus.cc:1261] T 83904952a5a541ecac508449e23e70dd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:33.710371 1132 consensus_queue.cc:1035] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:33.710522 31535 raft_consensus.cc:1261] T 83904952a5a541ecac508449e23e70dd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:33.711030 1132 consensus_queue.cc:1035] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:33.751114 1360 mvcc.cc:205] Tried to move back new op lower bound from 7023350553430691840 to 7023350551200534528. Current Snapshot: MvccSnapshot[applied={T|T < 7023350553430691840}]
INSERT report
rows total: 2000
time total: 273.353 ms
time per row: 0.136676 ms
I20240502 21:26:33.981237 31390 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalance_test_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:33.990584 31646 tablet_service.cc:1467] Processing CreateTablet for tablet 3d1bfb337f3c47a78c742cb665d27ffb (DEFAULT_TABLE table=rebalance_test_table_16 [id=ba6507441fb04f29a519bb81d3dba0e1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:33.990792 31516 tablet_service.cc:1467] Processing CreateTablet for tablet 3d1bfb337f3c47a78c742cb665d27ffb (DEFAULT_TABLE table=rebalance_test_table_16 [id=ba6507441fb04f29a519bb81d3dba0e1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:33.991007 31646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d1bfb337f3c47a78c742cb665d27ffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:33.991226 31516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d1bfb337f3c47a78c742cb665d27ffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:33.995465 1404 tablet_bootstrap.cc:492] T 3d1bfb337f3c47a78c742cb665d27ffb P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:33.995869 31906 tablet_service.cc:1467] Processing CreateTablet for tablet fe4e41225da24d7c91c6bf7733a5e067 (DEFAULT_TABLE table=rebalance_test_table_16 [id=ba6507441fb04f29a519bb81d3dba0e1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:33.995947 31776 tablet_service.cc:1467] Processing CreateTablet for tablet 3d1bfb337f3c47a78c742cb665d27ffb (DEFAULT_TABLE table=rebalance_test_table_16 [id=ba6507441fb04f29a519bb81d3dba0e1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:33.996315 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe4e41225da24d7c91c6bf7733a5e067. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:33.998525 1404 tablet_bootstrap.cc:654] T 3d1bfb337f3c47a78c742cb665d27ffb P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:33.996426 31776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d1bfb337f3c47a78c742cb665d27ffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:34.006024 31647 tablet_service.cc:1467] Processing CreateTablet for tablet fe4e41225da24d7c91c6bf7733a5e067 (DEFAULT_TABLE table=rebalance_test_table_16 [id=ba6507441fb04f29a519bb81d3dba0e1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:34.006392 31647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe4e41225da24d7c91c6bf7733a5e067. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:34.008767 31515 tablet_service.cc:1467] Processing CreateTablet for tablet fe4e41225da24d7c91c6bf7733a5e067 (DEFAULT_TABLE table=rebalance_test_table_16 [id=ba6507441fb04f29a519bb81d3dba0e1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:34.009171 31515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe4e41225da24d7c91c6bf7733a5e067. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:34.012893 1407 tablet_bootstrap.cc:492] T fe4e41225da24d7c91c6bf7733a5e067 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:34.013005 31777 tablet_service.cc:1467] Processing CreateTablet for tablet 80023f3b254c4369954ab92a085a0ec5 (DEFAULT_TABLE table=rebalance_test_table_16 [id=ba6507441fb04f29a519bb81d3dba0e1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:34.013567 31777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80023f3b254c4369954ab92a085a0ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:34.018008 31645 tablet_service.cc:1467] Processing CreateTablet for tablet 80023f3b254c4369954ab92a085a0ec5 (DEFAULT_TABLE table=rebalance_test_table_16 [id=ba6507441fb04f29a519bb81d3dba0e1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:34.018361 31645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80023f3b254c4369954ab92a085a0ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:34.017706 31906 tablet_service.cc:1467] Processing CreateTablet for tablet 80023f3b254c4369954ab92a085a0ec5 (DEFAULT_TABLE table=rebalance_test_table_16 [id=ba6507441fb04f29a519bb81d3dba0e1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:34.018698 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80023f3b254c4369954ab92a085a0ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:34.027732 1407 tablet_bootstrap.cc:654] T fe4e41225da24d7c91c6bf7733a5e067 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:34.033869 1404 tablet_bootstrap.cc:492] T 3d1bfb337f3c47a78c742cb665d27ffb P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:34.033927 1408 tablet_bootstrap.cc:492] T 3d1bfb337f3c47a78c742cb665d27ffb P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:34.034045 1404 ts_tablet_manager.cc:1367] T 3d1bfb337f3c47a78c742cb665d27ffb P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.007s
I20240502 21:26:34.034441 1405 tablet_bootstrap.cc:492] T 3d1bfb337f3c47a78c742cb665d27ffb P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:34.035928 1405 tablet_bootstrap.cc:654] T 3d1bfb337f3c47a78c742cb665d27ffb P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:34.036227 1404 raft_consensus.cc:348] T 3d1bfb337f3c47a78c742cb665d27ffb P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:34.036502 1404 raft_consensus.cc:374] T 3d1bfb337f3c47a78c742cb665d27ffb P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:34.036592 1404 raft_consensus.cc:729] T 3d1bfb337f3c47a78c742cb665d27ffb P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:34.036811 1404 consensus_queue.cc:260] T 3d1bfb337f3c47a78c742cb665d27ffb P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:34.037438 1404 ts_tablet_manager.cc:1398] T 3d1bfb337f3c47a78c742cb665d27ffb P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:34.037765 1405 tablet_bootstrap.cc:492] T 3d1bfb337f3c47a78c742cb665d27ffb P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:34.037884 1405 ts_tablet_manager.cc:1367] T 3d1bfb337f3c47a78c742cb665d27ffb P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:26:34.037894 1404 tablet_bootstrap.cc:492] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:34.038270 1408 tablet_bootstrap.cc:654] T 3d1bfb337f3c47a78c742cb665d27ffb P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:34.038499 1405 raft_consensus.cc:348] T 3d1bfb337f3c47a78c742cb665d27ffb P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:34.039815 1405 raft_consensus.cc:374] T 3d1bfb337f3c47a78c742cb665d27ffb P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:34.040597 1407 tablet_bootstrap.cc:492] T fe4e41225da24d7c91c6bf7733a5e067 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:34.040799 1407 ts_tablet_manager.cc:1367] T fe4e41225da24d7c91c6bf7733a5e067 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.002s
I20240502 21:26:34.041535 1404 tablet_bootstrap.cc:654] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:34.041541 1407 raft_consensus.cc:348] T fe4e41225da24d7c91c6bf7733a5e067 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:34.041831 1407 raft_consensus.cc:374] T fe4e41225da24d7c91c6bf7733a5e067 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:34.041951 1407 raft_consensus.cc:729] T fe4e41225da24d7c91c6bf7733a5e067 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:34.042243 1407 consensus_queue.cc:260] T fe4e41225da24d7c91c6bf7733a5e067 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:34.043943 1407 ts_tablet_manager.cc:1398] T fe4e41225da24d7c91c6bf7733a5e067 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:34.044173 1407 tablet_bootstrap.cc:492] T 80023f3b254c4369954ab92a085a0ec5 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:34.039966 1405 raft_consensus.cc:729] T 3d1bfb337f3c47a78c742cb665d27ffb P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:34.044723 1405 consensus_queue.cc:260] T 3d1bfb337f3c47a78c742cb665d27ffb P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:34.045570 1407 tablet_bootstrap.cc:654] T 80023f3b254c4369954ab92a085a0ec5 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:34.046063 1405 ts_tablet_manager.cc:1398] T 3d1bfb337f3c47a78c742cb665d27ffb P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 21:26:34.047389 1405 tablet_bootstrap.cc:492] T fe4e41225da24d7c91c6bf7733a5e067 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:34.048934 1405 tablet_bootstrap.cc:654] T fe4e41225da24d7c91c6bf7733a5e067 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:34.051913 1408 tablet_bootstrap.cc:492] T 3d1bfb337f3c47a78c742cb665d27ffb P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:34.052031 1404 tablet_bootstrap.cc:492] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:34.052147 1408 ts_tablet_manager.cc:1367] T 3d1bfb337f3c47a78c742cb665d27ffb P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.007s
I20240502 21:26:34.052182 1404 ts_tablet_manager.cc:1367] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.006s
I20240502 21:26:34.053059 1408 raft_consensus.cc:348] T 3d1bfb337f3c47a78c742cb665d27ffb P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:34.053359 1408 raft_consensus.cc:374] T 3d1bfb337f3c47a78c742cb665d27ffb P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:34.053531 1408 raft_consensus.cc:729] T 3d1bfb337f3c47a78c742cb665d27ffb P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:34.053849 1408 consensus_queue.cc:260] T 3d1bfb337f3c47a78c742cb665d27ffb P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:34.053866 1404 raft_consensus.cc:348] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:34.054044 1404 raft_consensus.cc:374] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:34.054401 1404 raft_consensus.cc:729] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:34.054572 1408 ts_tablet_manager.cc:1398] T 3d1bfb337f3c47a78c742cb665d27ffb P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:34.054638 1404 consensus_queue.cc:260] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:34.054883 1408 tablet_bootstrap.cc:492] T 80023f3b254c4369954ab92a085a0ec5 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:34.055423 1404 ts_tablet_manager.cc:1398] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:34.058068 1408 tablet_bootstrap.cc:654] T 80023f3b254c4369954ab92a085a0ec5 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:34.060400 1405 tablet_bootstrap.cc:492] T fe4e41225da24d7c91c6bf7733a5e067 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:34.060706 1405 ts_tablet_manager.cc:1367] T fe4e41225da24d7c91c6bf7733a5e067 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20240502 21:26:34.061673 1405 raft_consensus.cc:348] T fe4e41225da24d7c91c6bf7733a5e067 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:34.062057 1405 raft_consensus.cc:374] T fe4e41225da24d7c91c6bf7733a5e067 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:34.062183 1405 raft_consensus.cc:729] T fe4e41225da24d7c91c6bf7733a5e067 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:34.062942 1405 consensus_queue.cc:260] T fe4e41225da24d7c91c6bf7733a5e067 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:34.063575 1405 ts_tablet_manager.cc:1398] T fe4e41225da24d7c91c6bf7733a5e067 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:26:34.064195 1405 tablet_bootstrap.cc:492] T 80023f3b254c4369954ab92a085a0ec5 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:34.065660 1407 tablet_bootstrap.cc:492] T 80023f3b254c4369954ab92a085a0ec5 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:34.065768 1405 tablet_bootstrap.cc:654] T 80023f3b254c4369954ab92a085a0ec5 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:34.065812 1407 ts_tablet_manager.cc:1367] T 80023f3b254c4369954ab92a085a0ec5 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.009s
I20240502 21:26:34.066299 1408 tablet_bootstrap.cc:492] T 80023f3b254c4369954ab92a085a0ec5 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:34.066489 1408 ts_tablet_manager.cc:1367] T 80023f3b254c4369954ab92a085a0ec5 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.002s
I20240502 21:26:34.066535 1407 raft_consensus.cc:348] T 80023f3b254c4369954ab92a085a0ec5 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:34.066748 1407 raft_consensus.cc:374] T 80023f3b254c4369954ab92a085a0ec5 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:34.066893 1407 raft_consensus.cc:729] T 80023f3b254c4369954ab92a085a0ec5 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:34.067236 1408 raft_consensus.cc:348] T 80023f3b254c4369954ab92a085a0ec5 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:34.067451 1408 raft_consensus.cc:374] T 80023f3b254c4369954ab92a085a0ec5 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:34.067276 1407 consensus_queue.cc:260] T 80023f3b254c4369954ab92a085a0ec5 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:34.067550 1408 raft_consensus.cc:729] T 80023f3b254c4369954ab92a085a0ec5 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:34.067814 1408 consensus_queue.cc:260] T 80023f3b254c4369954ab92a085a0ec5 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:34.069469 1407 ts_tablet_manager.cc:1398] T 80023f3b254c4369954ab92a085a0ec5 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 21:26:34.069705 1408 ts_tablet_manager.cc:1398] T 80023f3b254c4369954ab92a085a0ec5 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:26:34.070673 1405 tablet_bootstrap.cc:492] T 80023f3b254c4369954ab92a085a0ec5 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:34.070789 1405 ts_tablet_manager.cc:1367] T 80023f3b254c4369954ab92a085a0ec5 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240502 21:26:34.071382 1405 raft_consensus.cc:348] T 80023f3b254c4369954ab92a085a0ec5 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:34.071573 1405 raft_consensus.cc:374] T 80023f3b254c4369954ab92a085a0ec5 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:34.071659 1405 raft_consensus.cc:729] T 80023f3b254c4369954ab92a085a0ec5 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:34.071870 1405 consensus_queue.cc:260] T 80023f3b254c4369954ab92a085a0ec5 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:34.073467 1405 ts_tablet_manager.cc:1398] T 80023f3b254c4369954ab92a085a0ec5 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:34.080579 1287 raft_consensus.cc:482] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:34.080757 1287 raft_consensus.cc:504] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:34.081473 1287 leader_election.cc:288] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:34.081831 31924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4e41225da24d7c91c6bf7733a5e067" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:34.081993 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4e41225da24d7c91c6bf7733a5e067" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:34.082103 31924 raft_consensus.cc:2454] T fe4e41225da24d7c91c6bf7733a5e067 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:34.082216 31666 raft_consensus.cc:2454] T fe4e41225da24d7c91c6bf7733a5e067 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:34.082455 31472 leader_election.cc:302] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:34.082787 1287 raft_consensus.cc:2781] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:34.082842 1000 raft_consensus.cc:482] T 3d1bfb337f3c47a78c742cb665d27ffb P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:34.082906 1287 raft_consensus.cc:482] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:34.083046 1287 raft_consensus.cc:3037] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:34.083022 1000 raft_consensus.cc:504] T 3d1bfb337f3c47a78c742cb665d27ffb P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:34.083688 1000 leader_election.cc:288] T 3d1bfb337f3c47a78c742cb665d27ffb P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:34.083897 31796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1bfb337f3c47a78c742cb665d27ffb" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:34.084098 31796 raft_consensus.cc:2454] T 3d1bfb337f3c47a78c742cb665d27ffb P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:34.084503 31471 leader_election.cc:302] T 3d1bfb337f3c47a78c742cb665d27ffb P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:34.084764 1000 raft_consensus.cc:2781] T 3d1bfb337f3c47a78c742cb665d27ffb P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:34.084759 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1bfb337f3c47a78c742cb665d27ffb" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:34.084868 1000 raft_consensus.cc:482] T 3d1bfb337f3c47a78c742cb665d27ffb P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:34.084964 1000 raft_consensus.cc:3037] T 3d1bfb337f3c47a78c742cb665d27ffb P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:34.084976 31666 raft_consensus.cc:2454] T 3d1bfb337f3c47a78c742cb665d27ffb P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:34.085520 1287 raft_consensus.cc:504] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:34.086282 1287 leader_election.cc:288] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Requested vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:34.086587 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4e41225da24d7c91c6bf7733a5e067" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:34.086747 31666 raft_consensus.cc:3037] T fe4e41225da24d7c91c6bf7733a5e067 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:34.087507 1000 raft_consensus.cc:504] T 3d1bfb337f3c47a78c742cb665d27ffb P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:34.088029 1000 leader_election.cc:288] T 3d1bfb337f3c47a78c742cb665d27ffb P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Requested vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:34.088451 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1bfb337f3c47a78c742cb665d27ffb" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:34.088642 31665 raft_consensus.cc:3037] T 3d1bfb337f3c47a78c742cb665d27ffb P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:34.088649 31924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4e41225da24d7c91c6bf7733a5e067" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:34.088809 31924 raft_consensus.cc:3037] T fe4e41225da24d7c91c6bf7733a5e067 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:34.089355 31666 raft_consensus.cc:2454] T fe4e41225da24d7c91c6bf7733a5e067 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:34.089702 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1bfb337f3c47a78c742cb665d27ffb" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:34.089877 31797 raft_consensus.cc:3037] T 3d1bfb337f3c47a78c742cb665d27ffb P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:34.090612 31665 raft_consensus.cc:2454] T 3d1bfb337f3c47a78c742cb665d27ffb P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:34.091249 31472 leader_election.cc:302] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:34.091547 31472 leader_election.cc:302] T 3d1bfb337f3c47a78c742cb665d27ffb P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:34.091759 1287 raft_consensus.cc:2781] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:34.091919 1287 raft_consensus.cc:686] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:26:34.092000 1417 raft_consensus.cc:2781] T 3d1bfb337f3c47a78c742cb665d27ffb P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:34.092350 31924 raft_consensus.cc:2454] T fe4e41225da24d7c91c6bf7733a5e067 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:34.092491 31797 raft_consensus.cc:2454] T 3d1bfb337f3c47a78c742cb665d27ffb P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:34.095518 1287 consensus_queue.cc:237] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:34.096640 1417 raft_consensus.cc:686] T 3d1bfb337f3c47a78c742cb665d27ffb P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:26:34.096997 1417 consensus_queue.cc:237] T 3d1bfb337f3c47a78c742cb665d27ffb P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:34.099093 31394 catalog_manager.cc:5484] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 0 to 1, leader changed from <none> to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:34.103706 31394 catalog_manager.cc:5484] T 3d1bfb337f3c47a78c742cb665d27ffb P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 0 to 1, leader changed from <none> to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:34.164661 1379 raft_consensus.cc:482] T 80023f3b254c4369954ab92a085a0ec5 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:34.164861 1379 raft_consensus.cc:504] T 80023f3b254c4369954ab92a085a0ec5 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:34.165676 1379 leader_election.cc:288] T 80023f3b254c4369954ab92a085a0ec5 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:34.166075 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80023f3b254c4369954ab92a085a0ec5" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:34.166082 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80023f3b254c4369954ab92a085a0ec5" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:34.166280 31665 raft_consensus.cc:2454] T 80023f3b254c4369954ab92a085a0ec5 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:34.166266 31797 raft_consensus.cc:2454] T 80023f3b254c4369954ab92a085a0ec5 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:34.166618 31863 leader_election.cc:302] T 80023f3b254c4369954ab92a085a0ec5 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:34.166844 1379 raft_consensus.cc:2781] T 80023f3b254c4369954ab92a085a0ec5 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:34.166934 1379 raft_consensus.cc:482] T 80023f3b254c4369954ab92a085a0ec5 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:34.167009 1379 raft_consensus.cc:3037] T 80023f3b254c4369954ab92a085a0ec5 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:34.168824 1379 raft_consensus.cc:504] T 80023f3b254c4369954ab92a085a0ec5 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:34.169353 1379 leader_election.cc:288] T 80023f3b254c4369954ab92a085a0ec5 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Requested vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:34.169740 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80023f3b254c4369954ab92a085a0ec5" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:34.169919 31797 raft_consensus.cc:3037] T 80023f3b254c4369954ab92a085a0ec5 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:34.169929 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80023f3b254c4369954ab92a085a0ec5" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:34.170122 31665 raft_consensus.cc:3037] T 80023f3b254c4369954ab92a085a0ec5 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:34.171747 31797 raft_consensus.cc:2454] T 80023f3b254c4369954ab92a085a0ec5 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:26:34.171851 31665 raft_consensus.cc:2454] T 80023f3b254c4369954ab92a085a0ec5 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:26:34.172145 31862 leader_election.cc:302] T 80023f3b254c4369954ab92a085a0ec5 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:34.172515 1379 raft_consensus.cc:2781] T 80023f3b254c4369954ab92a085a0ec5 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:34.172654 1379 raft_consensus.cc:686] T 80023f3b254c4369954ab92a085a0ec5 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Running, Role: LEADER
I20240502 21:26:34.172884 1379 consensus_queue.cc:237] T 80023f3b254c4369954ab92a085a0ec5 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:34.176535 31394 catalog_manager.cc:5484] T 80023f3b254c4369954ab92a085a0ec5 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132). New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:34.496652 1419 consensus_queue.cc:1035] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:34.502836 1287 consensus_queue.cc:1035] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:34.515956 1419 consensus_queue.cc:1035] T 3d1bfb337f3c47a78c742cb665d27ffb P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:34.521127 1417 consensus_queue.cc:1035] T 3d1bfb337f3c47a78c742cb665d27ffb P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:34.557654 1423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:34.558004 1423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 21:26:34.563822 1379 consensus_queue.cc:1035] T 80023f3b254c4369954ab92a085a0ec5 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:34.567544 1423 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:26:34.568670 1132 consensus_queue.cc:1035] T 80023f3b254c4369954ab92a085a0ec5 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_16'
INSERT report
rows total: 2000
time total: 262.155 ms
time per row: 0.131078 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 21:26:35.852030 31394 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalance_test_table_17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:35.860945 31776 tablet_service.cc:1467] Processing CreateTablet for tablet dd0198178da640688230aeb5a1106cb4 (DEFAULT_TABLE table=rebalance_test_table_17 [id=2e8e8260e3b64486b5712ad7289d1cc8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:35.861390 31776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd0198178da640688230aeb5a1106cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:35.861804 31516 tablet_service.cc:1467] Processing CreateTablet for tablet 62f083e9641042c990efcfb1c4104de5 (DEFAULT_TABLE table=rebalance_test_table_17 [id=2e8e8260e3b64486b5712ad7289d1cc8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:35.862195 31516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62f083e9641042c990efcfb1c4104de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:35.862154 31514 tablet_service.cc:1467] Processing CreateTablet for tablet 822bdadec3e24409a103906196d54e58 (DEFAULT_TABLE table=rebalance_test_table_17 [id=2e8e8260e3b64486b5712ad7289d1cc8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:35.862538 31514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 822bdadec3e24409a103906196d54e58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:35.864784 31515 tablet_service.cc:1467] Processing CreateTablet for tablet dd0198178da640688230aeb5a1106cb4 (DEFAULT_TABLE table=rebalance_test_table_17 [id=2e8e8260e3b64486b5712ad7289d1cc8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:35.865037 31515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd0198178da640688230aeb5a1106cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:35.867532 1491 tablet_bootstrap.cc:492] T dd0198178da640688230aeb5a1106cb4 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:35.868996 1491 tablet_bootstrap.cc:654] T dd0198178da640688230aeb5a1106cb4 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:35.869199 31906 tablet_service.cc:1467] Processing CreateTablet for tablet dd0198178da640688230aeb5a1106cb4 (DEFAULT_TABLE table=rebalance_test_table_17 [id=2e8e8260e3b64486b5712ad7289d1cc8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:35.869647 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd0198178da640688230aeb5a1106cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:35.872920 31905 tablet_service.cc:1467] Processing CreateTablet for tablet 62f083e9641042c990efcfb1c4104de5 (DEFAULT_TABLE table=rebalance_test_table_17 [id=2e8e8260e3b64486b5712ad7289d1cc8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:35.873296 31905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62f083e9641042c990efcfb1c4104de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:35.874330 31646 tablet_service.cc:1467] Processing CreateTablet for tablet 822bdadec3e24409a103906196d54e58 (DEFAULT_TABLE table=rebalance_test_table_17 [id=2e8e8260e3b64486b5712ad7289d1cc8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:35.874706 31646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 822bdadec3e24409a103906196d54e58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:35.877142 31777 tablet_service.cc:1467] Processing CreateTablet for tablet 822bdadec3e24409a103906196d54e58 (DEFAULT_TABLE table=rebalance_test_table_17 [id=2e8e8260e3b64486b5712ad7289d1cc8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:35.877573 31777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 822bdadec3e24409a103906196d54e58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:35.884107 31776 tablet_service.cc:1467] Processing CreateTablet for tablet 62f083e9641042c990efcfb1c4104de5 (DEFAULT_TABLE table=rebalance_test_table_17 [id=2e8e8260e3b64486b5712ad7289d1cc8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:35.884793 31776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62f083e9641042c990efcfb1c4104de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:35.888689 1495 tablet_bootstrap.cc:492] T dd0198178da640688230aeb5a1106cb4 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:35.889921 1491 tablet_bootstrap.cc:492] T dd0198178da640688230aeb5a1106cb4 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:35.890051 1491 ts_tablet_manager.cc:1367] T dd0198178da640688230aeb5a1106cb4 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.023s user 0.013s sys 0.007s
I20240502 21:26:35.890945 1495 tablet_bootstrap.cc:654] T dd0198178da640688230aeb5a1106cb4 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:35.891072 1491 raft_consensus.cc:348] T dd0198178da640688230aeb5a1106cb4 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:35.891295 1491 raft_consensus.cc:374] T dd0198178da640688230aeb5a1106cb4 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:35.891386 1491 raft_consensus.cc:729] T dd0198178da640688230aeb5a1106cb4 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:35.891569 1491 consensus_queue.cc:260] T dd0198178da640688230aeb5a1106cb4 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:35.892086 1491 ts_tablet_manager.cc:1398] T dd0198178da640688230aeb5a1106cb4 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:35.892582 1491 tablet_bootstrap.cc:492] T 822bdadec3e24409a103906196d54e58 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:35.892876 1494 tablet_bootstrap.cc:492] T 822bdadec3e24409a103906196d54e58 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:35.893975 1491 tablet_bootstrap.cc:654] T 822bdadec3e24409a103906196d54e58 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:35.895915 1494 tablet_bootstrap.cc:654] T 822bdadec3e24409a103906196d54e58 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:35.896696 1495 tablet_bootstrap.cc:492] T dd0198178da640688230aeb5a1106cb4 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:35.896888 1495 ts_tablet_manager.cc:1367] T dd0198178da640688230aeb5a1106cb4 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20240502 21:26:35.897763 1495 raft_consensus.cc:348] T dd0198178da640688230aeb5a1106cb4 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:35.899134 1495 raft_consensus.cc:374] T dd0198178da640688230aeb5a1106cb4 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:35.899413 1492 tablet_bootstrap.cc:492] T 62f083e9641042c990efcfb1c4104de5 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:35.900525 1495 raft_consensus.cc:729] T dd0198178da640688230aeb5a1106cb4 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:35.900849 1495 consensus_queue.cc:260] T dd0198178da640688230aeb5a1106cb4 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:35.901307 1491 tablet_bootstrap.cc:492] T 822bdadec3e24409a103906196d54e58 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:35.901446 1491 ts_tablet_manager.cc:1367] T 822bdadec3e24409a103906196d54e58 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.007s
I20240502 21:26:35.901551 1495 ts_tablet_manager.cc:1398] T dd0198178da640688230aeb5a1106cb4 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:26:35.901986 1491 raft_consensus.cc:348] T 822bdadec3e24409a103906196d54e58 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:35.902069 1492 tablet_bootstrap.cc:654] T 62f083e9641042c990efcfb1c4104de5 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:35.902146 1491 raft_consensus.cc:374] T 822bdadec3e24409a103906196d54e58 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:35.902225 1491 raft_consensus.cc:729] T 822bdadec3e24409a103906196d54e58 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:35.902455 1491 consensus_queue.cc:260] T 822bdadec3e24409a103906196d54e58 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:35.903021 1491 ts_tablet_manager.cc:1398] T 822bdadec3e24409a103906196d54e58 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 21:26:35.903316 1495 tablet_bootstrap.cc:492] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:35.904294 1491 tablet_bootstrap.cc:492] T 62f083e9641042c990efcfb1c4104de5 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:35.904978 1495 tablet_bootstrap.cc:654] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:35.905696 1491 tablet_bootstrap.cc:654] T 62f083e9641042c990efcfb1c4104de5 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:35.907630 1494 tablet_bootstrap.cc:492] T 822bdadec3e24409a103906196d54e58 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:35.907826 1494 ts_tablet_manager.cc:1367] T 822bdadec3e24409a103906196d54e58 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20240502 21:26:35.908754 1494 raft_consensus.cc:348] T 822bdadec3e24409a103906196d54e58 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:35.910987 1494 raft_consensus.cc:374] T 822bdadec3e24409a103906196d54e58 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:35.911172 1494 raft_consensus.cc:729] T 822bdadec3e24409a103906196d54e58 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:35.911940 1494 consensus_queue.cc:260] T 822bdadec3e24409a103906196d54e58 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:35.913985 1494 ts_tablet_manager.cc:1398] T 822bdadec3e24409a103906196d54e58 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 21:26:35.914460 1491 tablet_bootstrap.cc:492] T 62f083e9641042c990efcfb1c4104de5 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:35.914590 1491 ts_tablet_manager.cc:1367] T 62f083e9641042c990efcfb1c4104de5 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20240502 21:26:35.915227 1491 raft_consensus.cc:348] T 62f083e9641042c990efcfb1c4104de5 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:35.915467 1491 raft_consensus.cc:374] T 62f083e9641042c990efcfb1c4104de5 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:35.915563 1491 raft_consensus.cc:729] T 62f083e9641042c990efcfb1c4104de5 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:35.915757 1491 consensus_queue.cc:260] T 62f083e9641042c990efcfb1c4104de5 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:35.916244 1491 ts_tablet_manager.cc:1398] T 62f083e9641042c990efcfb1c4104de5 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:35.920075 1495 tablet_bootstrap.cc:492] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:35.920091 1492 tablet_bootstrap.cc:492] T 62f083e9641042c990efcfb1c4104de5 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:35.920220 1492 ts_tablet_manager.cc:1367] T 62f083e9641042c990efcfb1c4104de5 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20240502 21:26:35.920378 1495 ts_tablet_manager.cc:1367] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.006s
I20240502 21:26:35.920948 1492 raft_consensus.cc:348] T 62f083e9641042c990efcfb1c4104de5 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:35.921185 1492 raft_consensus.cc:374] T 62f083e9641042c990efcfb1c4104de5 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:35.921157 1495 raft_consensus.cc:348] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:35.921293 1492 raft_consensus.cc:729] T 62f083e9641042c990efcfb1c4104de5 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:35.921402 1495 raft_consensus.cc:374] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:35.921509 1495 raft_consensus.cc:729] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:35.921748 1495 consensus_queue.cc:260] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:35.921837 1492 consensus_queue.cc:260] T 62f083e9641042c990efcfb1c4104de5 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:35.922325 1495 ts_tablet_manager.cc:1398] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:26:35.923555 1492 ts_tablet_manager.cc:1398] T 62f083e9641042c990efcfb1c4104de5 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:26:35.923820 1492 tablet_bootstrap.cc:492] T 822bdadec3e24409a103906196d54e58 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:35.925380 1492 tablet_bootstrap.cc:654] T 822bdadec3e24409a103906196d54e58 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:35.927421 1492 tablet_bootstrap.cc:492] T 822bdadec3e24409a103906196d54e58 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:35.927556 1492 ts_tablet_manager.cc:1367] T 822bdadec3e24409a103906196d54e58 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:26:35.928287 1492 raft_consensus.cc:348] T 822bdadec3e24409a103906196d54e58 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:35.928500 1492 raft_consensus.cc:374] T 822bdadec3e24409a103906196d54e58 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:35.928589 1492 raft_consensus.cc:729] T 822bdadec3e24409a103906196d54e58 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:35.928835 1492 consensus_queue.cc:260] T 822bdadec3e24409a103906196d54e58 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:35.929489 1492 ts_tablet_manager.cc:1398] T 822bdadec3e24409a103906196d54e58 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:35.929785 1492 tablet_bootstrap.cc:492] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:35.931178 1492 tablet_bootstrap.cc:654] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:35.933097 1492 tablet_bootstrap.cc:492] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:35.933279 1492 ts_tablet_manager.cc:1367] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:26:35.933935 1492 raft_consensus.cc:348] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:35.934123 1492 raft_consensus.cc:374] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:35.934214 1492 raft_consensus.cc:729] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:35.934386 1492 consensus_queue.cc:260] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:35.934834 1492 ts_tablet_manager.cc:1398] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:26:35.959487 1474 raft_consensus.cc:482] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:35.959654 1474 raft_consensus.cc:504] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:35.960335 1474 leader_election.cc:288] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:35.960598 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62f083e9641042c990efcfb1c4104de5" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:35.960831 31535 raft_consensus.cc:2454] T 62f083e9641042c990efcfb1c4104de5 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:35.960889 31796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62f083e9641042c990efcfb1c4104de5" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:35.961174 31796 raft_consensus.cc:2454] T 62f083e9641042c990efcfb1c4104de5 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:35.961243 31863 leader_election.cc:302] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:35.961578 1474 raft_consensus.cc:2781] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:35.961692 1474 raft_consensus.cc:482] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:35.961786 1474 raft_consensus.cc:3037] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:35.963776 1474 raft_consensus.cc:504] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:35.964627 1474 leader_election.cc:288] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Requested vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:35.964756 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62f083e9641042c990efcfb1c4104de5" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:26:35.964780 31796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62f083e9641042c990efcfb1c4104de5" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:35.964965 31796 raft_consensus.cc:3037] T 62f083e9641042c990efcfb1c4104de5 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:35.964959 31535 raft_consensus.cc:3037] T 62f083e9641042c990efcfb1c4104de5 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:35.966718 31796 raft_consensus.cc:2454] T 62f083e9641042c990efcfb1c4104de5 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:26:35.966743 31535 raft_consensus.cc:2454] T 62f083e9641042c990efcfb1c4104de5 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:26:35.967049 31862 leader_election.cc:302] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:35.967392 1474 raft_consensus.cc:2781] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:35.967523 1474 raft_consensus.cc:686] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Running, Role: LEADER
I20240502 21:26:35.967770 1474 consensus_queue.cc:237] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:35.970839 31394 catalog_manager.cc:5484] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132). New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:35.984382 1476 raft_consensus.cc:482] T 822bdadec3e24409a103906196d54e58 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:35.984534 1476 raft_consensus.cc:504] T 822bdadec3e24409a103906196d54e58 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:35.985105 1476 leader_election.cc:288] T 822bdadec3e24409a103906196d54e58 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:35.985414 31796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "822bdadec3e24409a103906196d54e58" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:35.985623 31796 raft_consensus.cc:2454] T 822bdadec3e24409a103906196d54e58 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:35.985607 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "822bdadec3e24409a103906196d54e58" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:35.985795 31666 raft_consensus.cc:2454] T 822bdadec3e24409a103906196d54e58 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:35.985945 31471 leader_election.cc:302] T 822bdadec3e24409a103906196d54e58 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:35.986223 1476 raft_consensus.cc:2781] T 822bdadec3e24409a103906196d54e58 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:35.986344 1476 raft_consensus.cc:482] T 822bdadec3e24409a103906196d54e58 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:35.986436 1476 raft_consensus.cc:3037] T 822bdadec3e24409a103906196d54e58 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:35.988662 1476 raft_consensus.cc:504] T 822bdadec3e24409a103906196d54e58 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:35.989497 1476 leader_election.cc:288] T 822bdadec3e24409a103906196d54e58 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Requested vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:35.989836 31796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "822bdadec3e24409a103906196d54e58" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:35.989885 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "822bdadec3e24409a103906196d54e58" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:35.989995 31796 raft_consensus.cc:3037] T 822bdadec3e24409a103906196d54e58 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:35.990053 31666 raft_consensus.cc:3037] T 822bdadec3e24409a103906196d54e58 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:35.990720 1476 raft_consensus.cc:482] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:35.990828 1476 raft_consensus.cc:504] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:35.991632 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd0198178da640688230aeb5a1106cb4" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:35.991809 31797 raft_consensus.cc:2454] T dd0198178da640688230aeb5a1106cb4 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:35.992184 31471 leader_election.cc:302] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:35.992285 31796 raft_consensus.cc:2454] T 822bdadec3e24409a103906196d54e58 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:35.992759 31666 raft_consensus.cc:2454] T 822bdadec3e24409a103906196d54e58 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:35.993161 31472 leader_election.cc:302] T 822bdadec3e24409a103906196d54e58 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:35.993875 1507 raft_consensus.cc:2781] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:35.993963 1507 raft_consensus.cc:482] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:35.994071 1507 raft_consensus.cc:3037] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:35.994256 1476 leader_election.cc:288] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:35.994434 1476 raft_consensus.cc:2781] T 822bdadec3e24409a103906196d54e58 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:35.994570 1476 raft_consensus.cc:686] T 822bdadec3e24409a103906196d54e58 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:26:35.994716 31924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd0198178da640688230aeb5a1106cb4" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:35.994931 31924 raft_consensus.cc:2454] T dd0198178da640688230aeb5a1106cb4 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:35.994833 1476 consensus_queue.cc:237] T 822bdadec3e24409a103906196d54e58 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:35.996882 1507 raft_consensus.cc:504] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:35.997454 1507 leader_election.cc:288] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Requested vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:35.997872 31924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd0198178da640688230aeb5a1106cb4" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:35.998060 31924 raft_consensus.cc:3037] T dd0198178da640688230aeb5a1106cb4 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:35.998427 31796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd0198178da640688230aeb5a1106cb4" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:35.998592 31796 raft_consensus.cc:3037] T dd0198178da640688230aeb5a1106cb4 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:36.001163 31796 raft_consensus.cc:2454] T dd0198178da640688230aeb5a1106cb4 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:36.001380 31924 raft_consensus.cc:2454] T dd0198178da640688230aeb5a1106cb4 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:36.001513 31471 leader_election.cc:302] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:36.001813 1506 raft_consensus.cc:2781] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:36.001935 1506 raft_consensus.cc:686] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:26:36.002223 1506 consensus_queue.cc:237] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:36.005810 31394 catalog_manager.cc:5484] T 822bdadec3e24409a103906196d54e58 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 0 to 1, leader changed from <none> to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:36.010250 31394 catalog_manager.cc:5484] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 0 to 1, leader changed from <none> to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } }
W20240502 21:26:36.348145 1510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:36.348392 1510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:36.356196 1510 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_17'
I20240502 21:26:36.489356 31924 raft_consensus.cc:1261] T dd0198178da640688230aeb5a1106cb4 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:36.489508 31797 raft_consensus.cc:1261] T dd0198178da640688230aeb5a1106cb4 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:36.490007 1506 consensus_queue.cc:1035] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:36.490271 1476 consensus_queue.cc:1035] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:36.493538 31796 raft_consensus.cc:1261] T 822bdadec3e24409a103906196d54e58 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:36.494566 31665 raft_consensus.cc:1261] T 822bdadec3e24409a103906196d54e58 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:36.495661 1476 consensus_queue.cc:1035] T 822bdadec3e24409a103906196d54e58 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:36.496145 1476 consensus_queue.cc:1035] T 822bdadec3e24409a103906196d54e58 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:36.508173 31795 raft_consensus.cc:1261] T 62f083e9641042c990efcfb1c4104de5 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:36.508908 1132 consensus_queue.cc:1035] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:36.518393 31535 raft_consensus.cc:1261] T 62f083e9641042c990efcfb1c4104de5 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:36.519079 1132 consensus_queue.cc:1035] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 279.707 ms
time per row: 0.139853 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 21:26:37.694459 31393 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalance_test_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:37.703411 31646 tablet_service.cc:1467] Processing CreateTablet for tablet a4575a0989ac4140aec6b2cdb1e92032 (DEFAULT_TABLE table=rebalance_test_table_18 [id=1fffea26d24e40e88ee6944f57c34b08]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:37.703593 31906 tablet_service.cc:1467] Processing CreateTablet for tablet a4575a0989ac4140aec6b2cdb1e92032 (DEFAULT_TABLE table=rebalance_test_table_18 [id=1fffea26d24e40e88ee6944f57c34b08]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:37.703866 31646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4575a0989ac4140aec6b2cdb1e92032. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:37.703991 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4575a0989ac4140aec6b2cdb1e92032. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:37.708274 31776 tablet_service.cc:1467] Processing CreateTablet for tablet 2ccfe8f0b1dd450db470891faebbff99 (DEFAULT_TABLE table=rebalance_test_table_18 [id=1fffea26d24e40e88ee6944f57c34b08]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:37.708663 31776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ccfe8f0b1dd450db470891faebbff99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:37.708735 31645 tablet_service.cc:1467] Processing CreateTablet for tablet 2ccfe8f0b1dd450db470891faebbff99 (DEFAULT_TABLE table=rebalance_test_table_18 [id=1fffea26d24e40e88ee6944f57c34b08]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:37.708938 31516 tablet_service.cc:1467] Processing CreateTablet for tablet a4575a0989ac4140aec6b2cdb1e92032 (DEFAULT_TABLE table=rebalance_test_table_18 [id=1fffea26d24e40e88ee6944f57c34b08]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:37.709049 31645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ccfe8f0b1dd450db470891faebbff99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:37.709360 31516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4575a0989ac4140aec6b2cdb1e92032. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:37.709422 31647 tablet_service.cc:1467] Processing CreateTablet for tablet c0b0e63252b34943984f1898e725e602 (DEFAULT_TABLE table=rebalance_test_table_18 [id=1fffea26d24e40e88ee6944f57c34b08]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:37.709784 31647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b0e63252b34943984f1898e725e602. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:37.713377 31905 tablet_service.cc:1467] Processing CreateTablet for tablet 2ccfe8f0b1dd450db470891faebbff99 (DEFAULT_TABLE table=rebalance_test_table_18 [id=1fffea26d24e40e88ee6944f57c34b08]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:37.713769 31905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ccfe8f0b1dd450db470891faebbff99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:37.713775 31514 tablet_service.cc:1467] Processing CreateTablet for tablet c0b0e63252b34943984f1898e725e602 (DEFAULT_TABLE table=rebalance_test_table_18 [id=1fffea26d24e40e88ee6944f57c34b08]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:37.714124 31514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b0e63252b34943984f1898e725e602. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:37.716840 31907 tablet_service.cc:1467] Processing CreateTablet for tablet c0b0e63252b34943984f1898e725e602 (DEFAULT_TABLE table=rebalance_test_table_18 [id=1fffea26d24e40e88ee6944f57c34b08]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:37.717263 31907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b0e63252b34943984f1898e725e602. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:37.724262 1587 tablet_bootstrap.cc:492] T a4575a0989ac4140aec6b2cdb1e92032 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:37.725908 1587 tablet_bootstrap.cc:654] T a4575a0989ac4140aec6b2cdb1e92032 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:37.726960 1586 tablet_bootstrap.cc:492] T 2ccfe8f0b1dd450db470891faebbff99 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:37.728513 1586 tablet_bootstrap.cc:654] T 2ccfe8f0b1dd450db470891faebbff99 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:37.733636 1586 tablet_bootstrap.cc:492] T 2ccfe8f0b1dd450db470891faebbff99 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:37.737535 1586 ts_tablet_manager.cc:1367] T 2ccfe8f0b1dd450db470891faebbff99 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.006s
I20240502 21:26:37.739015 1586 raft_consensus.cc:348] T 2ccfe8f0b1dd450db470891faebbff99 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:37.739391 1586 raft_consensus.cc:374] T 2ccfe8f0b1dd450db470891faebbff99 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:37.739940 1586 raft_consensus.cc:729] T 2ccfe8f0b1dd450db470891faebbff99 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:37.740236 1586 consensus_queue.cc:260] T 2ccfe8f0b1dd450db470891faebbff99 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:37.741093 1586 ts_tablet_manager.cc:1398] T 2ccfe8f0b1dd450db470891faebbff99 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:37.741613 1586 tablet_bootstrap.cc:492] T a4575a0989ac4140aec6b2cdb1e92032 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:37.743244 1591 tablet_bootstrap.cc:492] T 2ccfe8f0b1dd450db470891faebbff99 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:37.743749 1586 tablet_bootstrap.cc:654] T a4575a0989ac4140aec6b2cdb1e92032 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:37.744961 1591 tablet_bootstrap.cc:654] T 2ccfe8f0b1dd450db470891faebbff99 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:37.746534 1590 tablet_bootstrap.cc:492] T a4575a0989ac4140aec6b2cdb1e92032 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:37.748654 1590 tablet_bootstrap.cc:654] T a4575a0989ac4140aec6b2cdb1e92032 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:37.750908 1591 tablet_bootstrap.cc:492] T 2ccfe8f0b1dd450db470891faebbff99 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:37.751076 1591 ts_tablet_manager.cc:1367] T 2ccfe8f0b1dd450db470891faebbff99 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 21:26:37.752750 1591 raft_consensus.cc:348] T 2ccfe8f0b1dd450db470891faebbff99 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:37.753832 1590 tablet_bootstrap.cc:492] T a4575a0989ac4140aec6b2cdb1e92032 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:37.753957 1591 raft_consensus.cc:374] T 2ccfe8f0b1dd450db470891faebbff99 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:37.754071 1591 raft_consensus.cc:729] T 2ccfe8f0b1dd450db470891faebbff99 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:37.754305 1591 consensus_queue.cc:260] T 2ccfe8f0b1dd450db470891faebbff99 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:37.755641 1590 ts_tablet_manager.cc:1367] T a4575a0989ac4140aec6b2cdb1e92032 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20240502 21:26:37.755977 1591 ts_tablet_manager.cc:1398] T 2ccfe8f0b1dd450db470891faebbff99 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 21:26:37.756883 1590 raft_consensus.cc:348] T a4575a0989ac4140aec6b2cdb1e92032 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:37.757112 1590 raft_consensus.cc:374] T a4575a0989ac4140aec6b2cdb1e92032 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:37.757206 1587 tablet_bootstrap.cc:492] T a4575a0989ac4140aec6b2cdb1e92032 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:37.757263 1590 raft_consensus.cc:729] T a4575a0989ac4140aec6b2cdb1e92032 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:37.757416 1587 ts_tablet_manager.cc:1367] T a4575a0989ac4140aec6b2cdb1e92032 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.033s user 0.008s sys 0.012s
I20240502 21:26:37.757843 1590 consensus_queue.cc:260] T a4575a0989ac4140aec6b2cdb1e92032 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:37.758206 1587 raft_consensus.cc:348] T a4575a0989ac4140aec6b2cdb1e92032 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:37.758395 1587 raft_consensus.cc:374] T a4575a0989ac4140aec6b2cdb1e92032 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:37.758495 1587 raft_consensus.cc:729] T a4575a0989ac4140aec6b2cdb1e92032 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:37.758677 1587 consensus_queue.cc:260] T a4575a0989ac4140aec6b2cdb1e92032 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:37.759160 1587 ts_tablet_manager.cc:1398] T a4575a0989ac4140aec6b2cdb1e92032 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:26:37.761579 1587 tablet_bootstrap.cc:492] T c0b0e63252b34943984f1898e725e602 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:37.763037 1587 tablet_bootstrap.cc:654] T c0b0e63252b34943984f1898e725e602 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:37.766849 1590 ts_tablet_manager.cc:1398] T a4575a0989ac4140aec6b2cdb1e92032 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.011s user 0.000s sys 0.001s
I20240502 21:26:37.768136 1590 tablet_bootstrap.cc:492] T c0b0e63252b34943984f1898e725e602 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:37.769740 1590 tablet_bootstrap.cc:654] T c0b0e63252b34943984f1898e725e602 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:37.773849 1590 tablet_bootstrap.cc:492] T c0b0e63252b34943984f1898e725e602 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:37.774048 1590 ts_tablet_manager.cc:1367] T c0b0e63252b34943984f1898e725e602 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20240502 21:26:37.774672 1587 tablet_bootstrap.cc:492] T c0b0e63252b34943984f1898e725e602 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:37.774690 1586 tablet_bootstrap.cc:492] T a4575a0989ac4140aec6b2cdb1e92032 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:37.774818 1587 ts_tablet_manager.cc:1367] T c0b0e63252b34943984f1898e725e602 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.002s
I20240502 21:26:37.774855 1586 ts_tablet_manager.cc:1367] T a4575a0989ac4140aec6b2cdb1e92032 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.033s user 0.012s sys 0.017s
I20240502 21:26:37.774855 1590 raft_consensus.cc:348] T c0b0e63252b34943984f1898e725e602 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:37.775180 1590 raft_consensus.cc:374] T c0b0e63252b34943984f1898e725e602 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:37.775297 1590 raft_consensus.cc:729] T c0b0e63252b34943984f1898e725e602 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:37.775579 1586 raft_consensus.cc:348] T a4575a0989ac4140aec6b2cdb1e92032 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:37.775625 1587 raft_consensus.cc:348] T c0b0e63252b34943984f1898e725e602 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:37.775785 1586 raft_consensus.cc:374] T a4575a0989ac4140aec6b2cdb1e92032 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:37.775619 1590 consensus_queue.cc:260] T c0b0e63252b34943984f1898e725e602 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:37.775857 1586 raft_consensus.cc:729] T a4575a0989ac4140aec6b2cdb1e92032 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:37.775861 1587 raft_consensus.cc:374] T c0b0e63252b34943984f1898e725e602 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:37.775934 1587 raft_consensus.cc:729] T c0b0e63252b34943984f1898e725e602 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:37.776067 1586 consensus_queue.cc:260] T a4575a0989ac4140aec6b2cdb1e92032 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:37.776362 1587 consensus_queue.cc:260] T c0b0e63252b34943984f1898e725e602 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:37.776577 1586 ts_tablet_manager.cc:1398] T a4575a0989ac4140aec6b2cdb1e92032 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:37.776836 1586 tablet_bootstrap.cc:492] T c0b0e63252b34943984f1898e725e602 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:37.778286 1586 tablet_bootstrap.cc:654] T c0b0e63252b34943984f1898e725e602 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:37.782445 1587 ts_tablet_manager.cc:1398] T c0b0e63252b34943984f1898e725e602 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240502 21:26:37.784104 1590 ts_tablet_manager.cc:1398] T c0b0e63252b34943984f1898e725e602 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.010s user 0.000s sys 0.001s
I20240502 21:26:37.784320 1590 tablet_bootstrap.cc:492] T 2ccfe8f0b1dd450db470891faebbff99 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:37.785444 1586 tablet_bootstrap.cc:492] T c0b0e63252b34943984f1898e725e602 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:37.785573 1586 ts_tablet_manager.cc:1367] T c0b0e63252b34943984f1898e725e602 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20240502 21:26:37.785796 1590 tablet_bootstrap.cc:654] T 2ccfe8f0b1dd450db470891faebbff99 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:37.786163 1586 raft_consensus.cc:348] T c0b0e63252b34943984f1898e725e602 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:37.786327 1586 raft_consensus.cc:374] T c0b0e63252b34943984f1898e725e602 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:37.786448 1586 raft_consensus.cc:729] T c0b0e63252b34943984f1898e725e602 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:37.786662 1586 consensus_queue.cc:260] T c0b0e63252b34943984f1898e725e602 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:37.787199 1586 ts_tablet_manager.cc:1398] T c0b0e63252b34943984f1898e725e602 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:37.788818 1590 tablet_bootstrap.cc:492] T 2ccfe8f0b1dd450db470891faebbff99 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:37.788949 1590 ts_tablet_manager.cc:1367] T 2ccfe8f0b1dd450db470891faebbff99 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 21:26:37.789562 1590 raft_consensus.cc:348] T 2ccfe8f0b1dd450db470891faebbff99 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:37.789778 1590 raft_consensus.cc:374] T 2ccfe8f0b1dd450db470891faebbff99 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:37.789881 1590 raft_consensus.cc:729] T 2ccfe8f0b1dd450db470891faebbff99 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:37.790092 1590 consensus_queue.cc:260] T 2ccfe8f0b1dd450db470891faebbff99 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:37.790621 1590 ts_tablet_manager.cc:1398] T 2ccfe8f0b1dd450db470891faebbff99 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:26:37.804368 1580 raft_consensus.cc:482] T a4575a0989ac4140aec6b2cdb1e92032 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:37.804534 1580 raft_consensus.cc:504] T a4575a0989ac4140aec6b2cdb1e92032 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:37.805019 1580 leader_election.cc:288] T a4575a0989ac4140aec6b2cdb1e92032 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:37.805334 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4575a0989ac4140aec6b2cdb1e92032" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:37.805408 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4575a0989ac4140aec6b2cdb1e92032" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:37.805496 31666 raft_consensus.cc:2454] T a4575a0989ac4140aec6b2cdb1e92032 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:37.805595 31535 raft_consensus.cc:2454] T a4575a0989ac4140aec6b2cdb1e92032 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:37.805864 31863 leader_election.cc:302] T a4575a0989ac4140aec6b2cdb1e92032 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:37.806066 1580 raft_consensus.cc:2781] T a4575a0989ac4140aec6b2cdb1e92032 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:37.806160 1580 raft_consensus.cc:482] T a4575a0989ac4140aec6b2cdb1e92032 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:37.806238 1580 raft_consensus.cc:3037] T a4575a0989ac4140aec6b2cdb1e92032 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:37.808018 1580 raft_consensus.cc:504] T a4575a0989ac4140aec6b2cdb1e92032 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:37.808581 1580 leader_election.cc:288] T a4575a0989ac4140aec6b2cdb1e92032 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Requested vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:37.808907 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4575a0989ac4140aec6b2cdb1e92032" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:37.809062 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4575a0989ac4140aec6b2cdb1e92032" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:26:37.809142 31666 raft_consensus.cc:3037] T a4575a0989ac4140aec6b2cdb1e92032 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:37.809249 31535 raft_consensus.cc:3037] T a4575a0989ac4140aec6b2cdb1e92032 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:37.811066 31535 raft_consensus.cc:2454] T a4575a0989ac4140aec6b2cdb1e92032 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:26:37.811188 31666 raft_consensus.cc:2454] T a4575a0989ac4140aec6b2cdb1e92032 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:26:37.811573 31863 leader_election.cc:302] T a4575a0989ac4140aec6b2cdb1e92032 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:37.811815 1580 raft_consensus.cc:2781] T a4575a0989ac4140aec6b2cdb1e92032 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:37.811985 1580 raft_consensus.cc:686] T a4575a0989ac4140aec6b2cdb1e92032 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Running, Role: LEADER
I20240502 21:26:37.812342 1580 consensus_queue.cc:237] T a4575a0989ac4140aec6b2cdb1e92032 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:37.815757 31390 catalog_manager.cc:5484] T a4575a0989ac4140aec6b2cdb1e92032 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132). New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:37.855566 1505 raft_consensus.cc:482] T c0b0e63252b34943984f1898e725e602 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:37.855679 1505 raft_consensus.cc:504] T c0b0e63252b34943984f1898e725e602 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:37.856123 1505 leader_election.cc:288] T c0b0e63252b34943984f1898e725e602 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:37.856510 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b0e63252b34943984f1898e725e602" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:37.856607 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b0e63252b34943984f1898e725e602" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:37.856736 31925 raft_consensus.cc:2454] T c0b0e63252b34943984f1898e725e602 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:26:37.856809 31535 raft_consensus.cc:2454] T c0b0e63252b34943984f1898e725e602 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:26:37.857182 31603 leader_election.cc:302] T c0b0e63252b34943984f1898e725e602 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:37.857463 1505 raft_consensus.cc:2781] T c0b0e63252b34943984f1898e725e602 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:37.857578 1505 raft_consensus.cc:482] T c0b0e63252b34943984f1898e725e602 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:37.857662 1505 raft_consensus.cc:3037] T c0b0e63252b34943984f1898e725e602 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:37.859323 1505 raft_consensus.cc:504] T c0b0e63252b34943984f1898e725e602 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:37.859742 1505 leader_election.cc:288] T c0b0e63252b34943984f1898e725e602 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Requested vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:37.860065 31924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b0e63252b34943984f1898e725e602" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:37.860172 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b0e63252b34943984f1898e725e602" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:26:37.860229 31924 raft_consensus.cc:3037] T c0b0e63252b34943984f1898e725e602 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:37.860368 31535 raft_consensus.cc:3037] T c0b0e63252b34943984f1898e725e602 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:37.862030 31924 raft_consensus.cc:2454] T c0b0e63252b34943984f1898e725e602 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:37.862332 31535 raft_consensus.cc:2454] T c0b0e63252b34943984f1898e725e602 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:37.862363 31603 leader_election.cc:302] T c0b0e63252b34943984f1898e725e602 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:37.862627 1505 raft_consensus.cc:2781] T c0b0e63252b34943984f1898e725e602 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:37.863018 1505 raft_consensus.cc:686] T c0b0e63252b34943984f1898e725e602 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Becoming Leader. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Running, Role: LEADER
I20240502 21:26:37.863265 1505 consensus_queue.cc:237] T c0b0e63252b34943984f1898e725e602 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:37.866554 31390 catalog_manager.cc:5484] T c0b0e63252b34943984f1898e725e602 P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130). New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:38.182374 1580 raft_consensus.cc:482] T 2ccfe8f0b1dd450db470891faebbff99 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:38.182601 1580 raft_consensus.cc:504] T 2ccfe8f0b1dd450db470891faebbff99 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:38.183290 1580 leader_election.cc:288] T 2ccfe8f0b1dd450db470891faebbff99 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:38.183691 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccfe8f0b1dd450db470891faebbff99" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:38.183919 31666 raft_consensus.cc:2454] T 2ccfe8f0b1dd450db470891faebbff99 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:38.183835 31795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccfe8f0b1dd450db470891faebbff99" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:38.184073 31795 raft_consensus.cc:2454] T 2ccfe8f0b1dd450db470891faebbff99 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:38.184267 31863 leader_election.cc:302] T 2ccfe8f0b1dd450db470891faebbff99 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:38.184587 1580 raft_consensus.cc:2781] T 2ccfe8f0b1dd450db470891faebbff99 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:38.184689 1580 raft_consensus.cc:482] T 2ccfe8f0b1dd450db470891faebbff99 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:38.184789 1580 raft_consensus.cc:3037] T 2ccfe8f0b1dd450db470891faebbff99 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:38.187249 1580 raft_consensus.cc:504] T 2ccfe8f0b1dd450db470891faebbff99 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:38.187778 1580 leader_election.cc:288] T 2ccfe8f0b1dd450db470891faebbff99 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Requested vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:38.188087 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccfe8f0b1dd450db470891faebbff99" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:38.188294 31666 raft_consensus.cc:3037] T 2ccfe8f0b1dd450db470891faebbff99 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:38.188239 31795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccfe8f0b1dd450db470891faebbff99" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:38.188426 31795 raft_consensus.cc:3037] T 2ccfe8f0b1dd450db470891faebbff99 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:38.191155 31795 raft_consensus.cc:2454] T 2ccfe8f0b1dd450db470891faebbff99 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:26:38.191171 31666 raft_consensus.cc:2454] T 2ccfe8f0b1dd450db470891faebbff99 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:26:38.191573 31863 leader_election.cc:302] T 2ccfe8f0b1dd450db470891faebbff99 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:38.191880 1580 raft_consensus.cc:2781] T 2ccfe8f0b1dd450db470891faebbff99 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:38.192044 1580 raft_consensus.cc:686] T 2ccfe8f0b1dd450db470891faebbff99 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Running, Role: LEADER
I20240502 21:26:38.192284 1580 consensus_queue.cc:237] T 2ccfe8f0b1dd450db470891faebbff99 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:38.195497 31390 catalog_manager.cc:5484] T 2ccfe8f0b1dd450db470891faebbff99 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132). New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:38.368983 1580 consensus_queue.cc:1035] T a4575a0989ac4140aec6b2cdb1e92032 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:38.376366 1545 consensus_queue.cc:1035] T a4575a0989ac4140aec6b2cdb1e92032 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:38.423198 1505 consensus_queue.cc:1035] T c0b0e63252b34943984f1898e725e602 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:38.428040 1612 consensus_queue.cc:1035] T c0b0e63252b34943984f1898e725e602 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:26:38.596763 1580 consensus_queue.cc:1035] T 2ccfe8f0b1dd450db470891faebbff99 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:38.603143 1545 consensus_queue.cc:1035] T 2ccfe8f0b1dd450db470891faebbff99 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:38.627115 1605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:38.628202 1605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:38.636291 1605 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_18'
INSERT report
rows total: 2000
time total: 302.22 ms
time per row: 0.15111 ms
I20240502 21:26:39.072554 31390 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalance_test_table_19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:39.080286 31776 tablet_service.cc:1467] Processing CreateTablet for tablet 043f33cb58084c3ebbce4893d1b63fbc (DEFAULT_TABLE table=rebalance_test_table_19 [id=654ed3d4f1ef4492ac274c10f2ba1f3d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:39.080802 31776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 043f33cb58084c3ebbce4893d1b63fbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:39.081223 31777 tablet_service.cc:1467] Processing CreateTablet for tablet f61e34e65429408e83a13c7bc743e78a (DEFAULT_TABLE table=rebalance_test_table_19 [id=654ed3d4f1ef4492ac274c10f2ba1f3d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:39.081478 31514 tablet_service.cc:1467] Processing CreateTablet for tablet 043f33cb58084c3ebbce4893d1b63fbc (DEFAULT_TABLE table=rebalance_test_table_19 [id=654ed3d4f1ef4492ac274c10f2ba1f3d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:39.081615 31777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f61e34e65429408e83a13c7bc743e78a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:39.081691 31645 tablet_service.cc:1467] Processing CreateTablet for tablet 043f33cb58084c3ebbce4893d1b63fbc (DEFAULT_TABLE table=rebalance_test_table_19 [id=654ed3d4f1ef4492ac274c10f2ba1f3d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:39.081961 31514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 043f33cb58084c3ebbce4893d1b63fbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:39.082135 31645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 043f33cb58084c3ebbce4893d1b63fbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:39.085009 31906 tablet_service.cc:1467] Processing CreateTablet for tablet f61e34e65429408e83a13c7bc743e78a (DEFAULT_TABLE table=rebalance_test_table_19 [id=654ed3d4f1ef4492ac274c10f2ba1f3d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:39.085440 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f61e34e65429408e83a13c7bc743e78a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:39.085515 31905 tablet_service.cc:1467] Processing CreateTablet for tablet 381ac1d714204cef86c8e1a73e24b863 (DEFAULT_TABLE table=rebalance_test_table_19 [id=654ed3d4f1ef4492ac274c10f2ba1f3d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:39.085852 31905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 381ac1d714204cef86c8e1a73e24b863. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:39.088757 31647 tablet_service.cc:1467] Processing CreateTablet for tablet f61e34e65429408e83a13c7bc743e78a (DEFAULT_TABLE table=rebalance_test_table_19 [id=654ed3d4f1ef4492ac274c10f2ba1f3d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:39.089223 31647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f61e34e65429408e83a13c7bc743e78a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:39.091378 1673 tablet_bootstrap.cc:492] T 043f33cb58084c3ebbce4893d1b63fbc P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:39.091653 31516 tablet_service.cc:1467] Processing CreateTablet for tablet 381ac1d714204cef86c8e1a73e24b863 (DEFAULT_TABLE table=rebalance_test_table_19 [id=654ed3d4f1ef4492ac274c10f2ba1f3d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:39.091972 31516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 381ac1d714204cef86c8e1a73e24b863. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:39.093281 1673 tablet_bootstrap.cc:654] T 043f33cb58084c3ebbce4893d1b63fbc P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:39.096881 31775 tablet_service.cc:1467] Processing CreateTablet for tablet 381ac1d714204cef86c8e1a73e24b863 (DEFAULT_TABLE table=rebalance_test_table_19 [id=654ed3d4f1ef4492ac274c10f2ba1f3d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:39.097329 31775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 381ac1d714204cef86c8e1a73e24b863. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:39.098484 1672 tablet_bootstrap.cc:492] T 043f33cb58084c3ebbce4893d1b63fbc P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:39.102613 1675 tablet_bootstrap.cc:492] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:39.104257 1675 tablet_bootstrap.cc:654] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:39.106563 1672 tablet_bootstrap.cc:654] T 043f33cb58084c3ebbce4893d1b63fbc P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:39.110913 1677 tablet_bootstrap.cc:492] T 381ac1d714204cef86c8e1a73e24b863 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:39.111651 1673 tablet_bootstrap.cc:492] T 043f33cb58084c3ebbce4893d1b63fbc P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:39.111862 1673 ts_tablet_manager.cc:1367] T 043f33cb58084c3ebbce4893d1b63fbc P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.004s
I20240502 21:26:39.112614 1675 tablet_bootstrap.cc:492] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:39.112787 1677 tablet_bootstrap.cc:654] T 381ac1d714204cef86c8e1a73e24b863 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:39.112790 1675 ts_tablet_manager.cc:1367] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.003s
I20240502 21:26:39.112998 1673 raft_consensus.cc:348] T 043f33cb58084c3ebbce4893d1b63fbc P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:39.113255 1673 raft_consensus.cc:374] T 043f33cb58084c3ebbce4893d1b63fbc P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:39.113344 1673 raft_consensus.cc:729] T 043f33cb58084c3ebbce4893d1b63fbc P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:39.113516 1675 raft_consensus.cc:348] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:39.113705 1675 raft_consensus.cc:374] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:39.113802 1675 raft_consensus.cc:729] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:39.113714 1673 consensus_queue.cc:260] T 043f33cb58084c3ebbce4893d1b63fbc P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:39.114090 1675 consensus_queue.cc:260] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:39.114873 1672 tablet_bootstrap.cc:492] T 043f33cb58084c3ebbce4893d1b63fbc P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:39.114933 1677 tablet_bootstrap.cc:492] T 381ac1d714204cef86c8e1a73e24b863 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:39.114989 1672 ts_tablet_manager.cc:1367] T 043f33cb58084c3ebbce4893d1b63fbc P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240502 21:26:39.115120 1677 ts_tablet_manager.cc:1367] T 381ac1d714204cef86c8e1a73e24b863 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 21:26:39.115865 1677 raft_consensus.cc:348] T 381ac1d714204cef86c8e1a73e24b863 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:39.116000 1675 ts_tablet_manager.cc:1398] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:26:39.116248 1675 tablet_bootstrap.cc:492] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:39.116499 1677 raft_consensus.cc:374] T 381ac1d714204cef86c8e1a73e24b863 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:39.116628 1677 raft_consensus.cc:729] T 381ac1d714204cef86c8e1a73e24b863 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:39.116860 1672 raft_consensus.cc:348] T 043f33cb58084c3ebbce4893d1b63fbc P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:39.116943 1677 consensus_queue.cc:260] T 381ac1d714204cef86c8e1a73e24b863 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:39.117127 1672 raft_consensus.cc:374] T 043f33cb58084c3ebbce4893d1b63fbc P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:39.117235 1672 raft_consensus.cc:729] T 043f33cb58084c3ebbce4893d1b63fbc P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:39.117532 1677 ts_tablet_manager.cc:1398] T 381ac1d714204cef86c8e1a73e24b863 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:39.117491 1672 consensus_queue.cc:260] T 043f33cb58084c3ebbce4893d1b63fbc P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:39.118065 1672 ts_tablet_manager.cc:1398] T 043f33cb58084c3ebbce4893d1b63fbc P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:39.118932 1673 ts_tablet_manager.cc:1398] T 043f33cb58084c3ebbce4893d1b63fbc P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:26:39.119169 1677 tablet_bootstrap.cc:492] T f61e34e65429408e83a13c7bc743e78a P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:39.119215 1672 tablet_bootstrap.cc:492] T f61e34e65429408e83a13c7bc743e78a P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:39.120631 1677 tablet_bootstrap.cc:654] T f61e34e65429408e83a13c7bc743e78a P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:39.120774 1673 tablet_bootstrap.cc:492] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:39.123067 1672 tablet_bootstrap.cc:654] T f61e34e65429408e83a13c7bc743e78a P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:39.124017 1673 tablet_bootstrap.cc:654] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:39.124611 1675 tablet_bootstrap.cc:654] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:39.129839 1672 tablet_bootstrap.cc:492] T f61e34e65429408e83a13c7bc743e78a P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:39.130115 1672 ts_tablet_manager.cc:1367] T f61e34e65429408e83a13c7bc743e78a P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240502 21:26:39.130721 1677 tablet_bootstrap.cc:492] T f61e34e65429408e83a13c7bc743e78a P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:39.130865 1677 ts_tablet_manager.cc:1367] T f61e34e65429408e83a13c7bc743e78a P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.009s
I20240502 21:26:39.131011 1672 raft_consensus.cc:348] T f61e34e65429408e83a13c7bc743e78a P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:39.131414 1672 raft_consensus.cc:374] T f61e34e65429408e83a13c7bc743e78a P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:39.131517 1672 raft_consensus.cc:729] T f61e34e65429408e83a13c7bc743e78a P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:39.131559 1677 raft_consensus.cc:348] T f61e34e65429408e83a13c7bc743e78a P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:39.131768 1677 raft_consensus.cc:374] T f61e34e65429408e83a13c7bc743e78a P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:39.131844 1677 raft_consensus.cc:729] T f61e34e65429408e83a13c7bc743e78a P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:39.131814 1672 consensus_queue.cc:260] T f61e34e65429408e83a13c7bc743e78a P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:39.132201 1677 consensus_queue.cc:260] T f61e34e65429408e83a13c7bc743e78a P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:39.132664 1672 ts_tablet_manager.cc:1398] T f61e34e65429408e83a13c7bc743e78a P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:39.133054 1672 tablet_bootstrap.cc:492] T 381ac1d714204cef86c8e1a73e24b863 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:39.134779 1672 tablet_bootstrap.cc:654] T 381ac1d714204cef86c8e1a73e24b863 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:39.135251 1677 ts_tablet_manager.cc:1398] T f61e34e65429408e83a13c7bc743e78a P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:26:39.137441 1675 tablet_bootstrap.cc:492] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:39.137444 1673 tablet_bootstrap.cc:492] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:39.137593 1673 ts_tablet_manager.cc:1367] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.009s
I20240502 21:26:39.137648 1675 ts_tablet_manager.cc:1367] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.006s
I20240502 21:26:39.137935 1672 tablet_bootstrap.cc:492] T 381ac1d714204cef86c8e1a73e24b863 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:39.138056 1672 ts_tablet_manager.cc:1367] T 381ac1d714204cef86c8e1a73e24b863 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:26:39.138459 1675 raft_consensus.cc:348] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:39.138689 1675 raft_consensus.cc:374] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:39.138630 1672 raft_consensus.cc:348] T 381ac1d714204cef86c8e1a73e24b863 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:39.138576 1673 raft_consensus.cc:348] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:39.138774 1672 raft_consensus.cc:374] T 381ac1d714204cef86c8e1a73e24b863 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:39.138803 1675 raft_consensus.cc:729] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:39.138865 1672 raft_consensus.cc:729] T 381ac1d714204cef86c8e1a73e24b863 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:39.138834 1673 raft_consensus.cc:374] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:39.138958 1673 raft_consensus.cc:729] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:39.139102 1672 consensus_queue.cc:260] T 381ac1d714204cef86c8e1a73e24b863 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:39.139122 1675 consensus_queue.cc:260] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:39.139209 1673 consensus_queue.cc:260] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:39.139614 1672 ts_tablet_manager.cc:1398] T 381ac1d714204cef86c8e1a73e24b863 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 21:26:39.139740 1675 ts_tablet_manager.cc:1398] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:26:39.140823 1673 ts_tablet_manager.cc:1398] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:26:39.243849 1505 raft_consensus.cc:482] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:39.244029 1505 raft_consensus.cc:504] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:39.244632 1505 leader_election.cc:288] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:39.244941 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043f33cb58084c3ebbce4893d1b63fbc" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:39.245033 31534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043f33cb58084c3ebbce4893d1b63fbc" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:39.245169 31797 raft_consensus.cc:2454] T 043f33cb58084c3ebbce4893d1b63fbc P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:26:39.245218 31534 raft_consensus.cc:2454] T 043f33cb58084c3ebbce4893d1b63fbc P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:26:39.245553 31602 leader_election.cc:302] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 08d7a0680d244b55b93d17f9ab57b84c; no voters:
I20240502 21:26:39.245791 1505 raft_consensus.cc:2781] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:39.245901 1505 raft_consensus.cc:482] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:39.245977 1505 raft_consensus.cc:3037] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:39.247788 1505 raft_consensus.cc:504] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:39.248258 1505 leader_election.cc:288] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Requested vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:39.248694 31534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043f33cb58084c3ebbce4893d1b63fbc" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:26:39.248672 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043f33cb58084c3ebbce4893d1b63fbc" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:39.248867 31534 raft_consensus.cc:3037] T 043f33cb58084c3ebbce4893d1b63fbc P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:39.248872 31797 raft_consensus.cc:3037] T 043f33cb58084c3ebbce4893d1b63fbc P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:39.251303 31797 raft_consensus.cc:2454] T 043f33cb58084c3ebbce4893d1b63fbc P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:39.251302 31534 raft_consensus.cc:2454] T 043f33cb58084c3ebbce4893d1b63fbc P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:39.251614 31602 leader_election.cc:302] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 08d7a0680d244b55b93d17f9ab57b84c; no voters:
I20240502 21:26:39.252224 1505 raft_consensus.cc:2781] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:39.252363 1505 raft_consensus.cc:686] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Becoming Leader. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Running, Role: LEADER
I20240502 21:26:39.252688 1505 consensus_queue.cc:237] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:39.255836 31394 catalog_manager.cc:5484] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130). New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:39.400625 1505 raft_consensus.cc:482] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:39.400797 1505 raft_consensus.cc:504] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:39.401360 1505 leader_election.cc:288] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:39.401680 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61e34e65429408e83a13c7bc743e78a" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:39.401916 31797 raft_consensus.cc:2454] T f61e34e65429408e83a13c7bc743e78a P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:26:39.401902 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61e34e65429408e83a13c7bc743e78a" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:39.402108 31925 raft_consensus.cc:2454] T f61e34e65429408e83a13c7bc743e78a P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:26:39.402249 31602 leader_election.cc:302] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 08d7a0680d244b55b93d17f9ab57b84c; no voters:
I20240502 21:26:39.402563 1505 raft_consensus.cc:2781] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:39.402665 1505 raft_consensus.cc:482] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:39.402824 1505 raft_consensus.cc:3037] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:39.404727 1505 raft_consensus.cc:504] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:39.405234 1505 leader_election.cc:288] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Requested vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:39.405565 31795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61e34e65429408e83a13c7bc743e78a" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:39.405587 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61e34e65429408e83a13c7bc743e78a" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:39.405718 31795 raft_consensus.cc:3037] T f61e34e65429408e83a13c7bc743e78a P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:39.405750 31925 raft_consensus.cc:3037] T f61e34e65429408e83a13c7bc743e78a P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:39.407747 31795 raft_consensus.cc:2454] T f61e34e65429408e83a13c7bc743e78a P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:39.407799 31925 raft_consensus.cc:2454] T f61e34e65429408e83a13c7bc743e78a P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:39.408094 31602 leader_election.cc:302] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 08d7a0680d244b55b93d17f9ab57b84c; no voters:
I20240502 21:26:39.408473 1505 raft_consensus.cc:2781] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:39.408630 1557 raft_consensus.cc:482] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:39.408726 1505 raft_consensus.cc:686] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Becoming Leader. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Running, Role: LEADER
I20240502 21:26:39.408780 1557 raft_consensus.cc:504] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:39.409044 1505 consensus_queue.cc:237] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:39.410049 31795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "381ac1d714204cef86c8e1a73e24b863" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:39.410151 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "381ac1d714204cef86c8e1a73e24b863" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:39.410272 31795 raft_consensus.cc:2454] T 381ac1d714204cef86c8e1a73e24b863 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:39.410324 31925 raft_consensus.cc:2454] T 381ac1d714204cef86c8e1a73e24b863 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:39.410506 1557 leader_election.cc:288] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:39.410634 31471 leader_election.cc:302] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:39.410871 1557 raft_consensus.cc:2781] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:39.410996 1557 raft_consensus.cc:482] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:39.411144 1557 raft_consensus.cc:3037] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:39.413801 31394 catalog_manager.cc:5484] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130). New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:39.414139 1557 raft_consensus.cc:504] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:39.414654 1557 leader_election.cc:288] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Requested vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:39.415167 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "381ac1d714204cef86c8e1a73e24b863" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:39.415176 31795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "381ac1d714204cef86c8e1a73e24b863" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:39.415315 31925 raft_consensus.cc:3037] T 381ac1d714204cef86c8e1a73e24b863 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:39.415331 31795 raft_consensus.cc:3037] T 381ac1d714204cef86c8e1a73e24b863 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:39.417366 31925 raft_consensus.cc:2454] T 381ac1d714204cef86c8e1a73e24b863 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:39.417779 31795 raft_consensus.cc:2454] T 381ac1d714204cef86c8e1a73e24b863 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:39.418257 31471 leader_election.cc:302] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:39.418515 1557 raft_consensus.cc:2781] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:39.418704 1557 raft_consensus.cc:686] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:26:39.418929 1557 consensus_queue.cc:237] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:39.421887 31394 catalog_manager.cc:5484] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 0 to 1, leader changed from <none> to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:39.654680 1505 consensus_queue.cc:1035] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:39.660486 1666 consensus_queue.cc:1035] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:39.793565 1688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:39.793818 1688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:39.800611 1688 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_19'
I20240502 21:26:39.872092 1666 consensus_queue.cc:1035] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:39.935189 1505 consensus_queue.cc:1035] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:39.956506 1557 consensus_queue.cc:1035] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:39.967121 1557 consensus_queue.cc:1035] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:39.987063 1714 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.24.226.190:43531, user_credentials={real_user=slave}} blocked reactor thread for 54844.5us
INSERT report
rows total: 2000
time total: 327.434 ms
time per row: 0.163717 ms
I20240502 21:26:40.285809 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.133:39863
--local_ip_for_outbound_sockets=127.24.226.133
--tserver_master_addrs=127.24.226.190:43531
--webserver_port=44979
--webserver_interface=127.24.226.133
--builtin_ntp_servers=127.24.226.148:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 21:26:40.603120 1752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:40.603513 1752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:40.603664 1752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:40.611196 1752 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:40.611295 1752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:40.611439 1752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.133
I20240502 21:26:40.620028 1752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.133:39863
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.24.226.133
--webserver_port=44979
--tserver_master_addrs=127.24.226.190:43531
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.133
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:40.621183 1752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:40.622759 1752 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:40.633430 1758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:40.633601 1760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:40.634183 1757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:40.634912 1752 server_base.cc:1034] running on GCE node
I20240502 21:26:40.635380 1752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:40.636123 1752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:40.637315 1752 hybrid_clock.cc:648] HybridClock initialized: now 1714685200637266 us; error 26 us; skew 500 ppm
I20240502 21:26:40.637578 1752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:40.639436 1752 webserver.cc:460] Webserver started at http://127.24.226.133:44979/ using document root <none> and password file <none>
I20240502 21:26:40.640007 1752 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:40.640123 1752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:40.644634 1752 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:26:40.646883 1766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:40.647910 1752 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:40.648041 1752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "1e04157b98dd4fad98858c290b73c3db"
format_stamp: "Formatted at 2024-05-02 21:26:05 on dist-test-slave-41k1"
I20240502 21:26:40.648753 1752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:40.669306 1752 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:40.669982 1752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:40.670212 1752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:40.670975 1752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:40.672498 1752 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:40.672583 1752 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:40.672694 1752 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:40.672747 1752 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:40.717257 1752 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.133:39863
I20240502 21:26:40.717350 1878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.133:39863 every 8 connection(s)
I20240502 21:26:40.718639 1752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:26:40.725598 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 1752
I20240502 21:26:40.726253 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.134:43753
--local_ip_for_outbound_sockets=127.24.226.134
--tserver_master_addrs=127.24.226.190:43531
--webserver_port=37149
--webserver_interface=127.24.226.134
--builtin_ntp_servers=127.24.226.148:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:26:40.730616 1879 heartbeater.cc:347] Connected to a master server at 127.24.226.190:43531
I20240502 21:26:40.730861 1879 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:40.731564 1879 heartbeater.cc:511] Master 127.24.226.190:43531 requested a full tablet report, sending...
I20240502 21:26:40.732867 31392 ts_manager.cc:194] Re-registered known tserver with Master: 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:26:40.733687 31392 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.133:38955
W20240502 21:26:41.046547 1883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:41.046818 1883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:41.046943 1883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:41.054961 1883 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:41.055061 1883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:41.055215 1883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.134
I20240502 21:26:41.063495 1883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.134:43753
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.24.226.134
--webserver_port=37149
--tserver_master_addrs=127.24.226.190:43531
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.134
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:41.064826 1883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:41.066432 1883 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:41.078466 1890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:41.078603 1889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:41.079401 1892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:41.079850 1883 server_base.cc:1034] running on GCE node
I20240502 21:26:41.080412 1883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:41.081305 1883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:41.082489 1883 hybrid_clock.cc:648] HybridClock initialized: now 1714685201082450 us; error 30 us; skew 500 ppm
I20240502 21:26:41.082732 1883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:41.084823 1883 webserver.cc:460] Webserver started at http://127.24.226.134:37149/ using document root <none> and password file <none>
I20240502 21:26:41.085425 1883 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:41.085525 1883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:41.089887 1883 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 21:26:41.092825 1898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:41.094072 1883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 21:26:41.094208 1883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "9dec645b473f40578e726f3e50e219bf"
format_stamp: "Formatted at 2024-05-02 21:26:05 on dist-test-slave-41k1"
I20240502 21:26:41.094871 1883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:41.106480 1883 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:41.107074 1883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:41.107300 1883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:41.108050 1883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:41.109400 1883 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:41.109464 1883 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:41.109542 1883 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:41.109582 1883 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:41.140928 1883 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.134:43753
I20240502 21:26:41.140974 2010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.134:43753 every 8 connection(s)
I20240502 21:26:41.142297 1883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 21:26:41.146668 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 1883
I20240502 21:26:41.147197 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.135:38569
--local_ip_for_outbound_sockets=127.24.226.135
--tserver_master_addrs=127.24.226.190:43531
--webserver_port=34145
--webserver_interface=127.24.226.135
--builtin_ntp_servers=127.24.226.148:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:26:41.154564 2011 heartbeater.cc:347] Connected to a master server at 127.24.226.190:43531
I20240502 21:26:41.154822 2011 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:41.155515 2011 heartbeater.cc:511] Master 127.24.226.190:43531 requested a full tablet report, sending...
I20240502 21:26:41.157012 31393 ts_manager.cc:194] Re-registered known tserver with Master: 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:26:41.158072 31393 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.134:35753
W20240502 21:26:41.471657 2014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:41.471880 2014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:41.472002 2014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:41.478668 2014 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:41.478761 2014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:41.478888 2014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.135
I20240502 21:26:41.487325 2014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.135:38569
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.24.226.135
--webserver_port=34145
--tserver_master_addrs=127.24.226.190:43531
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.135
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:41.488644 2014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:41.490262 2014 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:41.500782 2021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:41.500960 2023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:41.501516 2020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:41.502085 2014 server_base.cc:1034] running on GCE node
I20240502 21:26:41.502578 2014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:41.503417 2014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:41.504609 2014 hybrid_clock.cc:648] HybridClock initialized: now 1714685201504564 us; error 33 us; skew 500 ppm
I20240502 21:26:41.504902 2014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:41.507292 2014 webserver.cc:460] Webserver started at http://127.24.226.135:34145/ using document root <none> and password file <none>
I20240502 21:26:41.507917 2014 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:41.508028 2014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:41.513680 2014 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 21:26:41.516628 2029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:41.518029 2014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 21:26:41.518186 2014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
format_stamp: "Formatted at 2024-05-02 21:26:05 on dist-test-slave-41k1"
I20240502 21:26:41.518908 2014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:41.558796 2014 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:41.559607 2014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:41.559903 2014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:41.560873 2014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:41.562172 2014 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:41.562245 2014 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:41.562364 2014 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:41.562415 2014 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:41.602815 2014 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.135:38569
I20240502 21:26:41.602850 2142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.135:38569 every 8 connection(s)
I20240502 21:26:41.604188 2014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 21:26:41.610710 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 2014
I20240502 21:26:41.611382 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.136:34907
--local_ip_for_outbound_sockets=127.24.226.136
--tserver_master_addrs=127.24.226.190:43531
--webserver_port=40265
--webserver_interface=127.24.226.136
--builtin_ntp_servers=127.24.226.148:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:26:41.616848 2143 heartbeater.cc:347] Connected to a master server at 127.24.226.190:43531
I20240502 21:26:41.617105 2143 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:41.617997 2143 heartbeater.cc:511] Master 127.24.226.190:43531 requested a full tablet report, sending...
I20240502 21:26:41.619414 31393 ts_manager.cc:194] Re-registered known tserver with Master: 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:41.620117 31393 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.135:57041
I20240502 21:26:41.735675 1879 heartbeater.cc:503] Master 127.24.226.190:43531 was elected leader, sending a full tablet report...
W20240502 21:26:41.956406 2146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:41.956737 2146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:26:41.956864 2146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:41.964380 2146 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:26:41.964517 2146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:26:41.964648 2146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.136
I20240502 21:26:41.973649 2146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.136:34907
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.24.226.136
--webserver_port=40265
--tserver_master_addrs=127.24.226.190:43531
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.136
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:26:41.974867 2146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:26:41.976675 2146 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:26:41.991278 2151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:41.992702 2152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:26:41.996628 2154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:26:41.997844 2146 server_base.cc:1034] running on GCE node
I20240502 21:26:41.998454 2146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:26:41.999635 2146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:26:42.000852 2146 hybrid_clock.cc:648] HybridClock initialized: now 1714685202000784 us; error 40 us; skew 500 ppm
I20240502 21:26:42.001147 2146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:26:42.003720 2146 webserver.cc:460] Webserver started at http://127.24.226.136:40265/ using document root <none> and password file <none>
I20240502 21:26:42.004645 2146 fs_manager.cc:359] Metadata directory not provided
I20240502 21:26:42.004817 2146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:26:42.010116 2146 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20240502 21:26:42.013242 2160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:42.014704 2146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 21:26:42.014887 2146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "d1406e92aec84ac585304e57bd37a40d"
format_stamp: "Formatted at 2024-05-02 21:26:06 on dist-test-slave-41k1"
I20240502 21:26:42.015760 2146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:26:42.038654 2146 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:26:42.039393 2146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:26:42.039705 2146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:26:42.040736 2146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:26:42.042451 2146 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:26:42.042560 2146 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:42.042682 2146 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:26:42.042758 2146 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:26:42.086894 2146 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.136:34907
I20240502 21:26:42.086920 2275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.136:34907 every 8 connection(s)
I20240502 21:26:42.088192 2146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 21:26:42.094260 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 2146
I20240502 21:26:42.104117 2276 heartbeater.cc:347] Connected to a master server at 127.24.226.190:43531
I20240502 21:26:42.104454 2276 heartbeater.cc:464] Registering TS with master...
I20240502 21:26:42.105248 2276 heartbeater.cc:511] Master 127.24.226.190:43531 requested a full tablet report, sending...
I20240502 21:26:42.105736 31392 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:26:42.106566 31393 ts_manager.cc:194] Re-registered known tserver with Master: d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:42.107338 31393 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.136:49365
I20240502 21:26:42.119874 31645 tablet_service.cc:1467] Processing CreateTablet for tablet 4bf20cecabdb4015a5f060e7a5d93ee8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ff278d6933464177987e75768846cf87]), partition=
I20240502 21:26:42.120242 31645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bf20cecabdb4015a5f060e7a5d93ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:42.126317 2292 tablet_bootstrap.cc:492] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:42.127703 2292 tablet_bootstrap.cc:654] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:42.132849 2210 tablet_service.cc:1467] Processing CreateTablet for tablet 4bf20cecabdb4015a5f060e7a5d93ee8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ff278d6933464177987e75768846cf87]), partition=
I20240502 21:26:42.133989 2210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bf20cecabdb4015a5f060e7a5d93ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:42.139724 2292 tablet_bootstrap.cc:492] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:42.140365 2292 ts_tablet_manager.cc:1367] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.003s
I20240502 21:26:42.142630 2292 raft_consensus.cc:348] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:42.142969 2292 raft_consensus.cc:374] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:42.143115 2292 raft_consensus.cc:729] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:42.145548 2292 consensus_queue.cc:260] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:42.146739 2292 ts_tablet_manager.cc:1398] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 21:26:42.148079 1946 tablet_service.cc:1467] Processing CreateTablet for tablet 4bf20cecabdb4015a5f060e7a5d93ee8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ff278d6933464177987e75768846cf87]), partition=
I20240502 21:26:42.149219 1946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bf20cecabdb4015a5f060e7a5d93ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:42.156616 2294 tablet_bootstrap.cc:492] T 4bf20cecabdb4015a5f060e7a5d93ee8 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:42.159960 2294 tablet_bootstrap.cc:654] T 4bf20cecabdb4015a5f060e7a5d93ee8 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:42.160522 2011 heartbeater.cc:503] Master 127.24.226.190:43531 was elected leader, sending a full tablet report...
I20240502 21:26:42.161903 2294 log.cc:824] T 4bf20cecabdb4015a5f060e7a5d93ee8 P d1406e92aec84ac585304e57bd37a40d: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:42.170382 2296 tablet_bootstrap.cc:492] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:42.172539 2294 tablet_bootstrap.cc:492] T 4bf20cecabdb4015a5f060e7a5d93ee8 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:42.172865 2294 ts_tablet_manager.cc:1367] T 4bf20cecabdb4015a5f060e7a5d93ee8 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.006s
I20240502 21:26:42.174765 2296 tablet_bootstrap.cc:654] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:42.175774 2296 log.cc:824] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 9dec645b473f40578e726f3e50e219bf: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:42.178141 2296 tablet_bootstrap.cc:492] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:26:42.178401 2296 ts_tablet_manager.cc:1367] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240502 21:26:42.181568 2294 raft_consensus.cc:348] T 4bf20cecabdb4015a5f060e7a5d93ee8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:42.181910 2294 raft_consensus.cc:374] T 4bf20cecabdb4015a5f060e7a5d93ee8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:42.182027 2294 raft_consensus.cc:729] T 4bf20cecabdb4015a5f060e7a5d93ee8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:42.182678 2294 consensus_queue.cc:260] T 4bf20cecabdb4015a5f060e7a5d93ee8 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:42.183872 2276 heartbeater.cc:503] Master 127.24.226.190:43531 was elected leader, sending a full tablet report...
I20240502 21:26:42.184386 2294 ts_tablet_manager.cc:1398] T 4bf20cecabdb4015a5f060e7a5d93ee8 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.011s user 0.008s sys 0.002s
I20240502 21:26:42.186877 2296 raft_consensus.cc:348] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:42.187361 2296 raft_consensus.cc:374] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:42.187467 2296 raft_consensus.cc:729] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:26:42.188146 2296 consensus_queue.cc:260] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:42.190019 2296 ts_tablet_manager.cc:1398] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.012s user 0.006s sys 0.005s
I20240502 21:26:42.217406 1735 raft_consensus.cc:482] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:42.217582 1735 raft_consensus.cc:504] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:42.219482 1735 leader_election.cc:288] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:42.226544 2231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bf20cecabdb4015a5f060e7a5d93ee8" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:26:42.227047 2231 raft_consensus.cc:2454] T 4bf20cecabdb4015a5f060e7a5d93ee8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:26:42.227664 31602 leader_election.cc:302] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:42.227974 1735 raft_consensus.cc:2781] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:42.228091 1735 raft_consensus.cc:482] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:42.228194 1735 raft_consensus.cc:3037] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:42.230414 1966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bf20cecabdb4015a5f060e7a5d93ee8" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:26:42.230876 1735 raft_consensus.cc:504] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:42.231014 1966 raft_consensus.cc:2454] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:26:42.231509 1735 leader_election.cc:288] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Requested vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:42.232023 1966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bf20cecabdb4015a5f060e7a5d93ee8" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf"
I20240502 21:26:42.232198 1966 raft_consensus.cc:3037] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:42.232626 2231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bf20cecabdb4015a5f060e7a5d93ee8" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:26:42.232801 2231 raft_consensus.cc:3037] T 4bf20cecabdb4015a5f060e7a5d93ee8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:42.235160 1966 raft_consensus.cc:2454] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:42.235720 31600 leader_election.cc:302] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:26:42.235870 2231 raft_consensus.cc:2454] T 4bf20cecabdb4015a5f060e7a5d93ee8 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:42.236009 1735 raft_consensus.cc:2781] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:42.236145 1735 raft_consensus.cc:686] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Becoming Leader. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Running, Role: LEADER
I20240502 21:26:42.236670 1735 consensus_queue.cc:237] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:42.239857 31393 catalog_manager.cc:5484] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130). New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } }
W20240502 21:26:42.340163 2277 tablet.cc:2369] T 4bf20cecabdb4015a5f060e7a5d93ee8 P d1406e92aec84ac585304e57bd37a40d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 21:26:42.353988 31713 tablet.cc:2369] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:42.359287 31393 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:26:42.367260 1966 raft_consensus.cc:1261] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:42.367396 2231 raft_consensus.cc:1261] T 4bf20cecabdb4015a5f060e7a5d93ee8 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:42.368170 1735 consensus_queue.cc:1035] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:42.368484 1732 consensus_queue.cc:1035] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:42.381240 31393 catalog_manager.cc:5576] TS d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907) does not have the latest schema for tablet 4bf20cecabdb4015a5f060e7a5d93ee8 (table rebalancer_dynamic_table [id=ff278d6933464177987e75768846cf87]). Expected version 1 got 0
I20240502 21:26:42.385934 2309 tablet.cc:1722] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:42.387176 2309 tablet.cc:1620] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada: MemRowSet was empty: no flush needed.
I20240502 21:26:42.387967 2306 mvcc.cc:205] Tried to move back new op lower bound from 7023350588888424448 to 7023350588363120640. Current Snapshot: MvccSnapshot[applied={T|T < 7023350588888424448}]
I20240502 21:26:42.388938 31392 catalog_manager.cc:5576] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753) does not have the latest schema for tablet 4bf20cecabdb4015a5f060e7a5d93ee8 (table rebalancer_dynamic_table [id=ff278d6933464177987e75768846cf87]). Expected version 1 got 0
I20240502 21:26:42.403092 31392 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=ff278d6933464177987e75768846cf87] alter complete (version 1)
I20240502 21:26:42.413445 2310 tablet.cc:1722] T 4bf20cecabdb4015a5f060e7a5d93ee8 P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:42.415004 2310 tablet.cc:1620] T 4bf20cecabdb4015a5f060e7a5d93ee8 P d1406e92aec84ac585304e57bd37a40d: MemRowSet was empty: no flush needed.
I20240502 21:26:42.416115 31392 catalog_manager.cc:5576] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753) does not have the latest schema for tablet 4bf20cecabdb4015a5f060e7a5d93ee8 (table rebalancer_dynamic_table [id=ff278d6933464177987e75768846cf87]). Expected version 1 got 0
I20240502 21:26:42.414489 2311 tablet.cc:1722] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:42.419109 2311 tablet.cc:1620] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 9dec645b473f40578e726f3e50e219bf: MemRowSet was empty: no flush needed.
I20240502 21:26:42.419992 31392 catalog_manager.cc:5576] TS d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907) does not have the latest schema for tablet 4bf20cecabdb4015a5f060e7a5d93ee8 (table rebalancer_dynamic_table [id=ff278d6933464177987e75768846cf87]). Expected version 1 got 0
I20240502 21:26:42.507508 31392 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:26:42.516655 2309 tablet.cc:1722] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:42.517619 2309 tablet.cc:1620] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada: MemRowSet was empty: no flush needed.
I20240502 21:26:42.519532 2310 tablet.cc:1722] T 4bf20cecabdb4015a5f060e7a5d93ee8 P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:42.519987 2310 tablet.cc:1620] T 4bf20cecabdb4015a5f060e7a5d93ee8 P d1406e92aec84ac585304e57bd37a40d: MemRowSet was empty: no flush needed.
I20240502 21:26:42.524473 2311 tablet.cc:1722] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:42.525928 2311 tablet.cc:1620] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 9dec645b473f40578e726f3e50e219bf: MemRowSet was empty: no flush needed.
I20240502 21:26:42.534585 31393 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=ff278d6933464177987e75768846cf87] alter complete (version 2)
I20240502 21:26:42.596642 31393 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240502 21:26:42.596521 31390 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_extra_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:42.596832 31393 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240502 21:26:42.606178 31516 tablet_service.cc:1467] Processing CreateTablet for tablet 90af70ade50047d79664e63328101666 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=9e962597ef7c41f493f1a41cd94c913b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:42.606649 31516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90af70ade50047d79664e63328101666. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:42.612345 31514 tablet_service.cc:1467] Processing CreateTablet for tablet 815b1fc784714c8cb4ba76707c345625 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=9e962597ef7c41f493f1a41cd94c913b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:42.612804 31514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 815b1fc784714c8cb4ba76707c345625. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:42.613282 2210 tablet_service.cc:1467] Processing CreateTablet for tablet 90af70ade50047d79664e63328101666 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=9e962597ef7c41f493f1a41cd94c913b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:42.613813 2210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90af70ade50047d79664e63328101666. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:42.618750 31775 tablet_service.cc:1467] Processing CreateTablet for tablet 815b1fc784714c8cb4ba76707c345625 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=9e962597ef7c41f493f1a41cd94c913b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:42.619156 31775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 815b1fc784714c8cb4ba76707c345625. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:42.620765 31776 tablet_service.cc:1467] Processing CreateTablet for tablet b9342dbe0497499ba4c238f2ed033293 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=9e962597ef7c41f493f1a41cd94c913b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:42.621081 31776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9342dbe0497499ba4c238f2ed033293. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:42.623389 2143 heartbeater.cc:503] Master 127.24.226.190:43531 was elected leader, sending a full tablet report...
I20240502 21:26:42.624671 31393 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 0af9582263b842de9f78784ce62cb716
I20240502 21:26:42.625401 31393 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 174d4efa03b5450ebe34170e9279a390
I20240502 21:26:42.625852 31393 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 9d89305df0454659b691ad23a0ff3392
I20240502 21:26:42.627420 31905 tablet_service.cc:1514] Processing DeleteTablet for tablet 0af9582263b842de9f78784ce62cb716 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:42 UTC) from {username='slave'} at 127.0.0.1:47824
I20240502 21:26:42.628353 2328 tablet_replica.cc:330] T 0af9582263b842de9f78784ce62cb716 P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:26:42.628425 31515 tablet_service.cc:1514] Processing DeleteTablet for tablet 174d4efa03b5450ebe34170e9279a390 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:42 UTC) from {username='slave'} at 127.0.0.1:45194
I20240502 21:26:42.628574 31513 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d89305df0454659b691ad23a0ff3392 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:42 UTC) from {username='slave'} at 127.0.0.1:45194
I20240502 21:26:42.628917 2328 raft_consensus.cc:2229] T 0af9582263b842de9f78784ce62cb716 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:42.628964 31906 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d89305df0454659b691ad23a0ff3392 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:42 UTC) from {username='slave'} at 127.0.0.1:47824
I20240502 21:26:42.629634 31645 tablet_service.cc:1514] Processing DeleteTablet for tablet 174d4efa03b5450ebe34170e9279a390 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:42 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:42.630679 31647 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d89305df0454659b691ad23a0ff3392 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:42 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:42.630764 31774 tablet_service.cc:1514] Processing DeleteTablet for tablet 174d4efa03b5450ebe34170e9279a390 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:42 UTC) from {username='slave'} at 127.0.0.1:51596
I20240502 21:26:42.631167 31777 tablet_service.cc:1514] Processing DeleteTablet for tablet 0af9582263b842de9f78784ce62cb716 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:42 UTC) from {username='slave'} at 127.0.0.1:51596
I20240502 21:26:42.632530 31514 tablet_service.cc:1514] Processing DeleteTablet for tablet 0af9582263b842de9f78784ce62cb716 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:42 UTC) from {username='slave'} at 127.0.0.1:45194
I20240502 21:26:42.637960 2328 raft_consensus.cc:2258] T 0af9582263b842de9f78784ce62cb716 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:42.639214 31390 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:26:42.640969 2328 ts_tablet_manager.cc:1875] T 0af9582263b842de9f78784ce62cb716 P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:42.642506 2330 tablet_replica.cc:330] T 174d4efa03b5450ebe34170e9279a390 P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:42.644484 2330 raft_consensus.cc:2229] T 174d4efa03b5450ebe34170e9279a390 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:42.646270 2330 raft_consensus.cc:2258] T 174d4efa03b5450ebe34170e9279a390 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:42.650015 2330 ts_tablet_manager.cc:1875] T 174d4efa03b5450ebe34170e9279a390 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:42.651366 2328 ts_tablet_manager.cc:1888] T 0af9582263b842de9f78784ce62cb716 P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:42.651575 2328 log.cc:1196] T 0af9582263b842de9f78784ce62cb716 P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/0af9582263b842de9f78784ce62cb716
I20240502 21:26:42.652143 2328 ts_tablet_manager.cc:1909] T 0af9582263b842de9f78784ce62cb716 P 2d50f33557944bb480ea32199d9ee4f1: Deleting consensus metadata
I20240502 21:26:42.654434 2328 tablet_replica.cc:330] T 9d89305df0454659b691ad23a0ff3392 P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:26:42.654845 2328 raft_consensus.cc:2229] T 9d89305df0454659b691ad23a0ff3392 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:42.655108 2328 raft_consensus.cc:2258] T 9d89305df0454659b691ad23a0ff3392 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:42.655371 2309 tablet.cc:1722] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:42.656044 2328 ts_tablet_manager.cc:1875] T 9d89305df0454659b691ad23a0ff3392 P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:42.660091 2311 tablet.cc:1722] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:42.663287 2332 tablet_bootstrap.cc:492] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:42.664489 2310 tablet.cc:1722] T 4bf20cecabdb4015a5f060e7a5d93ee8 P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:42.665352 2332 tablet_bootstrap.cc:654] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:42.667040 2331 tablet_replica.cc:330] T 174d4efa03b5450ebe34170e9279a390 P 08d7a0680d244b55b93d17f9ab57b84c: stopping tablet replica
I20240502 21:26:42.667551 2331 raft_consensus.cc:2229] T 174d4efa03b5450ebe34170e9279a390 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:42.668258 2328 ts_tablet_manager.cc:1888] T 9d89305df0454659b691ad23a0ff3392 P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:42.668493 2328 log.cc:1196] T 9d89305df0454659b691ad23a0ff3392 P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/9d89305df0454659b691ad23a0ff3392
I20240502 21:26:42.668634 2330 ts_tablet_manager.cc:1888] T 174d4efa03b5450ebe34170e9279a390 P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:42.669050 2331 raft_consensus.cc:2258] T 174d4efa03b5450ebe34170e9279a390 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:42.669395 2328 ts_tablet_manager.cc:1909] T 9d89305df0454659b691ad23a0ff3392 P 2d50f33557944bb480ea32199d9ee4f1: Deleting consensus metadata
I20240502 21:26:42.670519 2331 ts_tablet_manager.cc:1875] T 174d4efa03b5450ebe34170e9279a390 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:42.670895 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet 0af9582263b842de9f78784ce62cb716 (table rebalance_test_table_0 [id=d813d81d1b9645bab949b858e177664a]) successfully deleted
I20240502 21:26:42.671188 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet 9d89305df0454659b691ad23a0ff3392 (table rebalance_test_table_0 [id=d813d81d1b9645bab949b858e177664a]) successfully deleted
I20240502 21:26:42.673696 2330 log.cc:1196] T 174d4efa03b5450ebe34170e9279a390 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/174d4efa03b5450ebe34170e9279a390
I20240502 21:26:42.674829 2330 ts_tablet_manager.cc:1909] T 174d4efa03b5450ebe34170e9279a390 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:26:42.679526 2330 tablet_replica.cc:330] T 9d89305df0454659b691ad23a0ff3392 P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:42.679744 2330 raft_consensus.cc:2229] T 9d89305df0454659b691ad23a0ff3392 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:42.679841 31392 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table_new_name [id=ff278d6933464177987e75768846cf87] alter complete (version 3)
I20240502 21:26:42.680994 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet 174d4efa03b5450ebe34170e9279a390 (table rebalance_test_table_0 [id=d813d81d1b9645bab949b858e177664a]) successfully deleted
I20240502 21:26:42.682099 2330 raft_consensus.cc:2258] T 9d89305df0454659b691ad23a0ff3392 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:42.683398 2330 ts_tablet_manager.cc:1875] T 9d89305df0454659b691ad23a0ff3392 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:42.688387 2326 tablet_bootstrap.cc:492] T 90af70ade50047d79664e63328101666 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:42.689754 2339 tablet_bootstrap.cc:492] T 90af70ade50047d79664e63328101666 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:42.692731 2339 tablet_bootstrap.cc:654] T 90af70ade50047d79664e63328101666 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:42.694332 2326 tablet_bootstrap.cc:654] T 90af70ade50047d79664e63328101666 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:42.696637 2331 ts_tablet_manager.cc:1888] T 174d4efa03b5450ebe34170e9279a390 P 08d7a0680d244b55b93d17f9ab57b84c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:42.697978 2331 log.cc:1196] T 174d4efa03b5450ebe34170e9279a390 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/wals/174d4efa03b5450ebe34170e9279a390
I20240502 21:26:42.698634 2332 tablet_bootstrap.cc:492] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:42.698765 2332 ts_tablet_manager.cc:1367] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.036s user 0.000s sys 0.006s
I20240502 21:26:42.699054 2329 tablet_replica.cc:330] T 174d4efa03b5450ebe34170e9279a390 P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:42.699393 2332 raft_consensus.cc:348] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:42.699635 2332 raft_consensus.cc:374] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:42.699775 2332 raft_consensus.cc:729] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:42.700182 2329 raft_consensus.cc:2229] T 174d4efa03b5450ebe34170e9279a390 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:42.700350 2332 consensus_queue.cc:260] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:42.701025 2332 ts_tablet_manager.cc:1398] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:26:42.701314 2332 tablet_bootstrap.cc:492] T b9342dbe0497499ba4c238f2ed033293 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:42.702937 2330 ts_tablet_manager.cc:1888] T 9d89305df0454659b691ad23a0ff3392 P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:42.703627 2332 tablet_bootstrap.cc:654] T b9342dbe0497499ba4c238f2ed033293 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:42.705448 2331 ts_tablet_manager.cc:1909] T 174d4efa03b5450ebe34170e9279a390 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting consensus metadata
I20240502 21:26:42.707327 31381 catalog_manager.cc:4830] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): tablet 174d4efa03b5450ebe34170e9279a390 (table rebalance_test_table_0 [id=d813d81d1b9645bab949b858e177664a]) successfully deleted
I20240502 21:26:42.707895 2330 log.cc:1196] T 9d89305df0454659b691ad23a0ff3392 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/9d89305df0454659b691ad23a0ff3392
I20240502 21:26:42.707988 2331 tablet_replica.cc:330] T 0af9582263b842de9f78784ce62cb716 P 08d7a0680d244b55b93d17f9ab57b84c: stopping tablet replica
I20240502 21:26:42.708225 2331 raft_consensus.cc:2229] T 0af9582263b842de9f78784ce62cb716 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:42.708673 2331 raft_consensus.cc:2258] T 0af9582263b842de9f78784ce62cb716 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:42.709364 2331 ts_tablet_manager.cc:1875] T 0af9582263b842de9f78784ce62cb716 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:42.708701 2330 ts_tablet_manager.cc:1909] T 9d89305df0454659b691ad23a0ff3392 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:26:42.709899 2329 raft_consensus.cc:2258] T 174d4efa03b5450ebe34170e9279a390 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:42.710647 2339 tablet_bootstrap.cc:492] T 90af70ade50047d79664e63328101666 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:42.710825 2339 ts_tablet_manager.cc:1367] T 90af70ade50047d79664e63328101666 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.003s
I20240502 21:26:42.711409 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet 9d89305df0454659b691ad23a0ff3392 (table rebalance_test_table_0 [id=d813d81d1b9645bab949b858e177664a]) successfully deleted
I20240502 21:26:42.712090 2339 raft_consensus.cc:348] T 90af70ade50047d79664e63328101666 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:42.712460 2339 raft_consensus.cc:374] T 90af70ade50047d79664e63328101666 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:42.712715 2339 raft_consensus.cc:729] T 90af70ade50047d79664e63328101666 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:42.713299 2326 tablet_bootstrap.cc:492] T 90af70ade50047d79664e63328101666 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:42.713217 2339 consensus_queue.cc:260] T 90af70ade50047d79664e63328101666 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:42.713546 2326 ts_tablet_manager.cc:1367] T 90af70ade50047d79664e63328101666 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20240502 21:26:42.714730 2326 raft_consensus.cc:348] T 90af70ade50047d79664e63328101666 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:42.715096 2326 raft_consensus.cc:374] T 90af70ade50047d79664e63328101666 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:42.715243 2326 raft_consensus.cc:729] T 90af70ade50047d79664e63328101666 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:42.715503 2329 ts_tablet_manager.cc:1875] T 174d4efa03b5450ebe34170e9279a390 P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:42.716665 2332 tablet_bootstrap.cc:492] T b9342dbe0497499ba4c238f2ed033293 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:42.716717 2339 ts_tablet_manager.cc:1398] T 90af70ade50047d79664e63328101666 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:26:42.716931 2332 ts_tablet_manager.cc:1367] T b9342dbe0497499ba4c238f2ed033293 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.004s
I20240502 21:26:42.717561 2332 raft_consensus.cc:348] T b9342dbe0497499ba4c238f2ed033293 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:42.715504 2326 consensus_queue.cc:260] T 90af70ade50047d79664e63328101666 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:42.718223 2326 ts_tablet_manager.cc:1398] T 90af70ade50047d79664e63328101666 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:26:42.719437 2326 tablet_bootstrap.cc:492] T 815b1fc784714c8cb4ba76707c345625 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
W20240502 21:26:42.724035 31582 tablet.cc:2369] T 90af70ade50047d79664e63328101666 P a44f41b0b6644fb6aa29476aed8c964f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:42.723750 1813 tablet_service.cc:1467] Processing CreateTablet for tablet b9342dbe0497499ba4c238f2ed033293 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=9e962597ef7c41f493f1a41cd94c913b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:42.725319 1813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9342dbe0497499ba4c238f2ed033293. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:42.727988 1814 tablet_service.cc:1467] Processing CreateTablet for tablet 815b1fc784714c8cb4ba76707c345625 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=9e962597ef7c41f493f1a41cd94c913b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:42.728693 1814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 815b1fc784714c8cb4ba76707c345625. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:42.732100 2326 tablet_bootstrap.cc:654] T 815b1fc784714c8cb4ba76707c345625 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:42.733248 2332 raft_consensus.cc:374] T b9342dbe0497499ba4c238f2ed033293 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:42.733366 2332 raft_consensus.cc:729] T b9342dbe0497499ba4c238f2ed033293 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:42.734072 2332 consensus_queue.cc:260] T b9342dbe0497499ba4c238f2ed033293 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:42.736761 2332 ts_tablet_manager.cc:1398] T b9342dbe0497499ba4c238f2ed033293 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.020s user 0.000s sys 0.002s
I20240502 21:26:42.743531 2331 ts_tablet_manager.cc:1888] T 0af9582263b842de9f78784ce62cb716 P 08d7a0680d244b55b93d17f9ab57b84c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:42.743690 2331 log.cc:1196] T 0af9582263b842de9f78784ce62cb716 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/wals/0af9582263b842de9f78784ce62cb716
I20240502 21:26:42.743319 2077 tablet_service.cc:1467] Processing CreateTablet for tablet 90af70ade50047d79664e63328101666 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=9e962597ef7c41f493f1a41cd94c913b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:42.745298 2077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90af70ade50047d79664e63328101666. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:42.745380 2076 tablet_service.cc:1467] Processing CreateTablet for tablet b9342dbe0497499ba4c238f2ed033293 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=9e962597ef7c41f493f1a41cd94c913b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
W20240502 21:26:42.745544 31843 tablet.cc:2369] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:42.745815 2076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9342dbe0497499ba4c238f2ed033293. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:42.750016 2345 tablet_bootstrap.cc:492] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:42.755149 2329 ts_tablet_manager.cc:1888] T 174d4efa03b5450ebe34170e9279a390 P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:42.755434 2329 log.cc:1196] T 174d4efa03b5450ebe34170e9279a390 P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/174d4efa03b5450ebe34170e9279a390
I20240502 21:26:42.756191 2329 ts_tablet_manager.cc:1909] T 174d4efa03b5450ebe34170e9279a390 P a44f41b0b6644fb6aa29476aed8c964f: Deleting consensus metadata
I20240502 21:26:42.757993 2331 ts_tablet_manager.cc:1909] T 0af9582263b842de9f78784ce62cb716 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting consensus metadata
I20240502 21:26:42.759012 31381 catalog_manager.cc:4830] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): tablet 0af9582263b842de9f78784ce62cb716 (table rebalance_test_table_0 [id=d813d81d1b9645bab949b858e177664a]) successfully deleted
I20240502 21:26:42.759789 2326 tablet_bootstrap.cc:492] T 815b1fc784714c8cb4ba76707c345625 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:42.760044 2326 ts_tablet_manager.cc:1367] T 815b1fc784714c8cb4ba76707c345625 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.003s
I20240502 21:26:42.761137 2326 raft_consensus.cc:348] T 815b1fc784714c8cb4ba76707c345625 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:42.761467 2326 raft_consensus.cc:374] T 815b1fc784714c8cb4ba76707c345625 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:42.761617 2326 raft_consensus.cc:729] T 815b1fc784714c8cb4ba76707c345625 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:42.761922 2326 consensus_queue.cc:260] T 815b1fc784714c8cb4ba76707c345625 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:42.762619 2326 ts_tablet_manager.cc:1398] T 815b1fc784714c8cb4ba76707c345625 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:42.764585 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet 174d4efa03b5450ebe34170e9279a390 (table rebalance_test_table_0 [id=d813d81d1b9645bab949b858e177664a]) successfully deleted
I20240502 21:26:42.765067 2329 tablet_replica.cc:330] T 9d89305df0454659b691ad23a0ff3392 P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:42.765558 2329 raft_consensus.cc:2229] T 9d89305df0454659b691ad23a0ff3392 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:42.766940 2345 tablet_bootstrap.cc:654] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:42.768332 2347 tablet_bootstrap.cc:492] T b9342dbe0497499ba4c238f2ed033293 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:42.770912 2329 raft_consensus.cc:2258] T 9d89305df0454659b691ad23a0ff3392 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:42.771859 2329 ts_tablet_manager.cc:1875] T 9d89305df0454659b691ad23a0ff3392 P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:42.771994 2347 tablet_bootstrap.cc:654] T b9342dbe0497499ba4c238f2ed033293 P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:42.773155 2347 log.cc:824] T b9342dbe0497499ba4c238f2ed033293 P 1ade0354c44c4c37ae480c7a94b11e1b: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:42.774471 2345 log.cc:824] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db: Log is configured to *not* fsync() on all Append() calls
I20240502 21:26:42.779662 2329 ts_tablet_manager.cc:1888] T 9d89305df0454659b691ad23a0ff3392 P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:42.780663 2329 log.cc:1196] T 9d89305df0454659b691ad23a0ff3392 P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/9d89305df0454659b691ad23a0ff3392
I20240502 21:26:42.781683 2329 ts_tablet_manager.cc:1909] T 9d89305df0454659b691ad23a0ff3392 P a44f41b0b6644fb6aa29476aed8c964f: Deleting consensus metadata
I20240502 21:26:42.782778 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet 9d89305df0454659b691ad23a0ff3392 (table rebalance_test_table_0 [id=d813d81d1b9645bab949b858e177664a]) successfully deleted
I20240502 21:26:42.784250 2329 tablet_replica.cc:330] T 0af9582263b842de9f78784ce62cb716 P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:42.784967 2329 raft_consensus.cc:2229] T 0af9582263b842de9f78784ce62cb716 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:42.785339 2329 raft_consensus.cc:2258] T 0af9582263b842de9f78784ce62cb716 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:42.785548 2345 tablet_bootstrap.cc:492] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:42.785859 2345 ts_tablet_manager.cc:1367] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.036s user 0.011s sys 0.000s
I20240502 21:26:42.786172 2329 ts_tablet_manager.cc:1875] T 0af9582263b842de9f78784ce62cb716 P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:42.786626 31393 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:42.786824 31393 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:42.790148 31393 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 4bf20cecabdb4015a5f060e7a5d93ee8
I20240502 21:26:42.791134 1946 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bf20cecabdb4015a5f060e7a5d93ee8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:42 UTC) from {username='slave'} at 127.0.0.1:33002
I20240502 21:26:42.791281 2210 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bf20cecabdb4015a5f060e7a5d93ee8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:42 UTC) from {username='slave'} at 127.0.0.1:36984
I20240502 21:26:42.792115 31645 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bf20cecabdb4015a5f060e7a5d93ee8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:42 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:42.791961 2353 tablet_replica.cc:330] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 9dec645b473f40578e726f3e50e219bf: stopping tablet replica
I20240502 21:26:42.792598 2330 tablet_replica.cc:330] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:42.792795 2330 raft_consensus.cc:2229] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:42.792870 2353 raft_consensus.cc:2229] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:42.793206 2330 raft_consensus.cc:2258] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:42.793270 2353 raft_consensus.cc:2258] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:42.793502 1689 raft_consensus.cc:482] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:42.793656 1689 raft_consensus.cc:504] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:42.794464 1689 leader_election.cc:288] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:26:42.794832 2330 ts_tablet_manager.cc:1875] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:42.795238 31534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815b1fc784714c8cb4ba76707c345625" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:42.795114 2345 raft_consensus.cc:348] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:42.795439 31534 raft_consensus.cc:2454] T 815b1fc784714c8cb4ba76707c345625 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:42.795485 2345 raft_consensus.cc:374] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:42.795562 2345 raft_consensus.cc:729] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:42.796969 31733 leader_election.cc:302] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:42.797811 1689 raft_consensus.cc:2781] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:42.797943 1689 raft_consensus.cc:482] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:42.798030 1689 raft_consensus.cc:3037] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:42.801832 1689 raft_consensus.cc:504] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:42.804368 2330 ts_tablet_manager.cc:1888] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:42.804419 2347 tablet_bootstrap.cc:492] T b9342dbe0497499ba4c238f2ed033293 P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:42.804409 31534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815b1fc784714c8cb4ba76707c345625" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:26:42.804594 2330 log.cc:1196] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/4bf20cecabdb4015a5f060e7a5d93ee8
I20240502 21:26:42.804646 31534 raft_consensus.cc:3037] T 815b1fc784714c8cb4ba76707c345625 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:42.805930 2347 ts_tablet_manager.cc:1367] T b9342dbe0497499ba4c238f2ed033293 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.038s user 0.005s sys 0.006s
I20240502 21:26:42.806389 2330 ts_tablet_manager.cc:1909] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:26:42.807448 31534 raft_consensus.cc:2454] T 815b1fc784714c8cb4ba76707c345625 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:42.807581 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet 4bf20cecabdb4015a5f060e7a5d93ee8 (table rebalancer_dynamic_table_new_name [id=ff278d6933464177987e75768846cf87]) successfully deleted
I20240502 21:26:42.807870 31733 leader_election.cc:302] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:42.808148 1690 raft_consensus.cc:2781] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:42.809156 2329 ts_tablet_manager.cc:1888] T 0af9582263b842de9f78784ce62cb716 P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:42.809307 2329 log.cc:1196] T 0af9582263b842de9f78784ce62cb716 P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/0af9582263b842de9f78784ce62cb716
I20240502 21:26:42.809334 1689 leader_election.cc:288] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Requested vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:26:42.810096 2329 ts_tablet_manager.cc:1909] T 0af9582263b842de9f78784ce62cb716 P a44f41b0b6644fb6aa29476aed8c964f: Deleting consensus metadata
I20240502 21:26:42.811126 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet 0af9582263b842de9f78784ce62cb716 (table rebalance_test_table_0 [id=d813d81d1b9645bab949b858e177664a]) successfully deleted
I20240502 21:26:42.811743 2354 tablet_replica.cc:330] T 4bf20cecabdb4015a5f060e7a5d93ee8 P d1406e92aec84ac585304e57bd37a40d: stopping tablet replica
I20240502 21:26:42.812263 2354 raft_consensus.cc:2229] T 4bf20cecabdb4015a5f060e7a5d93ee8 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:42.815404 2354 raft_consensus.cc:2258] T 4bf20cecabdb4015a5f060e7a5d93ee8 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:42.815980 2353 ts_tablet_manager.cc:1875] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 9dec645b473f40578e726f3e50e219bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:42.817209 2354 ts_tablet_manager.cc:1875] T 4bf20cecabdb4015a5f060e7a5d93ee8 P d1406e92aec84ac585304e57bd37a40d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:42.818313 1690 raft_consensus.cc:686] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Becoming Leader. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Running, Role: LEADER
I20240502 21:26:42.818629 1690 consensus_queue.cc:237] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:42.823988 2345 consensus_queue.cc:260] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:42.824609 2356 raft_consensus.cc:482] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:42.824700 2353 ts_tablet_manager.cc:1888] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 9dec645b473f40578e726f3e50e219bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:42.824836 2353 log.cc:1196] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 9dec645b473f40578e726f3e50e219bf: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal/wals/4bf20cecabdb4015a5f060e7a5d93ee8
I20240502 21:26:42.825196 2354 ts_tablet_manager.cc:1888] T 4bf20cecabdb4015a5f060e7a5d93ee8 P d1406e92aec84ac585304e57bd37a40d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:42.825021 2356 raft_consensus.cc:504] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:42.825366 2354 log.cc:1196] T 4bf20cecabdb4015a5f060e7a5d93ee8 P d1406e92aec84ac585304e57bd37a40d: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal/wals/4bf20cecabdb4015a5f060e7a5d93ee8
I20240502 21:26:42.825774 1834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815b1fc784714c8cb4ba76707c345625" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:26:42.826088 2354 ts_tablet_manager.cc:1909] T 4bf20cecabdb4015a5f060e7a5d93ee8 P d1406e92aec84ac585304e57bd37a40d: Deleting consensus metadata
I20240502 21:26:42.826880 1833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815b1fc784714c8cb4ba76707c345625" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:26:42.827420 2353 ts_tablet_manager.cc:1909] T 4bf20cecabdb4015a5f060e7a5d93ee8 P 9dec645b473f40578e726f3e50e219bf: Deleting consensus metadata
W20240502 21:26:42.828671 31732 leader_election.cc:341] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:26:42.829056 31732 leader_election.cc:341] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:42.829357 31381 catalog_manager.cc:4830] TS d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): tablet 4bf20cecabdb4015a5f060e7a5d93ee8 (table rebalancer_dynamic_table_new_name [id=ff278d6933464177987e75768846cf87]) successfully deleted
I20240502 21:26:42.831784 31379 catalog_manager.cc:4830] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): tablet 4bf20cecabdb4015a5f060e7a5d93ee8 (table rebalancer_dynamic_table_new_name [id=ff278d6933464177987e75768846cf87]) successfully deleted
I20240502 21:26:42.832721 31393 catalog_manager.cc:5484] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: term changed from 0 to 1, leader changed from <none> to 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131). New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:42.834745 2356 leader_election.cc:288] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:42.835966 2345 ts_tablet_manager.cc:1398] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.050s user 0.009s sys 0.000s
I20240502 21:26:42.841063 2345 tablet_bootstrap.cc:492] T 815b1fc784714c8cb4ba76707c345625 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:42.843400 2345 tablet_bootstrap.cc:654] T 815b1fc784714c8cb4ba76707c345625 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:42.844913 2347 raft_consensus.cc:348] T b9342dbe0497499ba4c238f2ed033293 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:42.848461 2347 raft_consensus.cc:374] T b9342dbe0497499ba4c238f2ed033293 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:42.848618 2347 raft_consensus.cc:729] T b9342dbe0497499ba4c238f2ed033293 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:42.849570 2347 consensus_queue.cc:260] T b9342dbe0497499ba4c238f2ed033293 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:42.851555 2345 tablet_bootstrap.cc:492] T 815b1fc784714c8cb4ba76707c345625 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:42.851831 2345 ts_tablet_manager.cc:1367] T 815b1fc784714c8cb4ba76707c345625 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.004s
I20240502 21:26:42.853283 2345 raft_consensus.cc:348] T 815b1fc784714c8cb4ba76707c345625 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:42.853709 2345 raft_consensus.cc:374] T 815b1fc784714c8cb4ba76707c345625 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:42.853889 2345 raft_consensus.cc:729] T 815b1fc784714c8cb4ba76707c345625 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:42.854296 2345 consensus_queue.cc:260] T 815b1fc784714c8cb4ba76707c345625 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:42.857417 2345 ts_tablet_manager.cc:1398] T 815b1fc784714c8cb4ba76707c345625 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:26:42.858685 2097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9342dbe0497499ba4c238f2ed033293" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:42.859223 2097 raft_consensus.cc:2454] T b9342dbe0497499ba4c238f2ed033293 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 0.
I20240502 21:26:42.863708 2347 ts_tablet_manager.cc:1398] T b9342dbe0497499ba4c238f2ed033293 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.057s user 0.015s sys 0.006s
I20240502 21:26:42.859972 1769 leader_election.cc:302] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 1e04157b98dd4fad98858c290b73c3db; no voters:
I20240502 21:26:42.864229 2356 raft_consensus.cc:2781] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:42.864413 2356 raft_consensus.cc:482] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:42.864478 2347 tablet_bootstrap.cc:492] T 90af70ade50047d79664e63328101666 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:42.864548 2356 raft_consensus.cc:3037] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:42.866221 2347 tablet_bootstrap.cc:654] T 90af70ade50047d79664e63328101666 P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:42.867502 2356 raft_consensus.cc:504] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:42.868538 2097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9342dbe0497499ba4c238f2ed033293" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:26:42.868722 2097 raft_consensus.cc:3037] T b9342dbe0497499ba4c238f2ed033293 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:42.871026 2097 raft_consensus.cc:2454] T b9342dbe0497499ba4c238f2ed033293 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:26:42.871399 1769 leader_election.cc:302] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 1e04157b98dd4fad98858c290b73c3db; no voters:
I20240502 21:26:42.871675 2359 raft_consensus.cc:2781] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:42.873091 2356 leader_election.cc:288] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Requested vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:42.879390 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9342dbe0497499ba4c238f2ed033293" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:42.879575 31795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9342dbe0497499ba4c238f2ed033293" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:42.879655 31797 raft_consensus.cc:2454] T b9342dbe0497499ba4c238f2ed033293 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 0.
I20240502 21:26:42.884730 2359 raft_consensus.cc:686] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Becoming Leader. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Running, Role: LEADER
I20240502 21:26:42.885386 2359 consensus_queue.cc:237] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:42.888685 2347 tablet_bootstrap.cc:492] T 90af70ade50047d79664e63328101666 P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:42.888876 2347 ts_tablet_manager.cc:1367] T 90af70ade50047d79664e63328101666 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.004s
I20240502 21:26:42.889883 2347 raft_consensus.cc:348] T 90af70ade50047d79664e63328101666 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:42.890141 2347 raft_consensus.cc:374] T 90af70ade50047d79664e63328101666 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:42.890251 2347 raft_consensus.cc:729] T 90af70ade50047d79664e63328101666 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:42.890540 2347 consensus_queue.cc:260] T 90af70ade50047d79664e63328101666 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:42.891508 2347 ts_tablet_manager.cc:1398] T 90af70ade50047d79664e63328101666 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:42.893556 31392 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:26:42.897878 31393 catalog_manager.cc:5484] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e04157b98dd4fad98858c290b73c3db (127.24.226.133). New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:42.905109 2077 tablet_service.cc:1467] Processing CreateTablet for tablet e2006432b1c94d4f91cc91b213a28934 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bc87b3ebaa8c4f1b858fb14c03cb7504]), partition=
I20240502 21:26:42.905418 2077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2006432b1c94d4f91cc91b213a28934. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:42.905436 1946 tablet_service.cc:1467] Processing CreateTablet for tablet e2006432b1c94d4f91cc91b213a28934 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bc87b3ebaa8c4f1b858fb14c03cb7504]), partition=
I20240502 21:26:42.905736 1946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2006432b1c94d4f91cc91b213a28934. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:42.910081 1813 tablet_service.cc:1467] Processing CreateTablet for tablet e2006432b1c94d4f91cc91b213a28934 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bc87b3ebaa8c4f1b858fb14c03cb7504]), partition=
I20240502 21:26:42.910415 1813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2006432b1c94d4f91cc91b213a28934. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:42.913697 2347 tablet_bootstrap.cc:492] T e2006432b1c94d4f91cc91b213a28934 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:42.915930 2347 tablet_bootstrap.cc:654] T e2006432b1c94d4f91cc91b213a28934 P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:42.916231 2359 raft_consensus.cc:482] T 815b1fc784714c8cb4ba76707c345625 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:42.916568 2359 raft_consensus.cc:504] T 815b1fc784714c8cb4ba76707c345625 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:42.918180 31795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815b1fc784714c8cb4ba76707c345625" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:42.918979 2359 leader_election.cc:288] T 815b1fc784714c8cb4ba76707c345625 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:42.933846 2345 tablet_bootstrap.cc:492] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:42.935806 2345 tablet_bootstrap.cc:654] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:42.936653 2303 raft_consensus.cc:482] T 90af70ade50047d79664e63328101666 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:42.936787 2303 raft_consensus.cc:504] T 90af70ade50047d79664e63328101666 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:42.937520 2303 leader_election.cc:288] T 90af70ade50047d79664e63328101666 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:42.941172 2379 tablet_bootstrap.cc:492] T e2006432b1c94d4f91cc91b213a28934 P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:42.944070 2345 tablet_bootstrap.cc:492] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:42.944322 2345 ts_tablet_manager.cc:1367] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.003s
I20240502 21:26:42.945874 2345 raft_consensus.cc:348] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:42.946090 2345 raft_consensus.cc:374] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:42.946166 2345 raft_consensus.cc:729] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:42.946408 2345 consensus_queue.cc:260] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:42.947533 2345 ts_tablet_manager.cc:1398] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:26:42.947926 2347 tablet_bootstrap.cc:492] T e2006432b1c94d4f91cc91b213a28934 P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:42.948086 2347 ts_tablet_manager.cc:1367] T e2006432b1c94d4f91cc91b213a28934 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.004s
I20240502 21:26:42.948714 2347 raft_consensus.cc:348] T e2006432b1c94d4f91cc91b213a28934 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:42.949005 2347 raft_consensus.cc:374] T e2006432b1c94d4f91cc91b213a28934 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:42.949077 2347 raft_consensus.cc:729] T e2006432b1c94d4f91cc91b213a28934 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:42.949297 2347 consensus_queue.cc:260] T e2006432b1c94d4f91cc91b213a28934 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:42.949811 2347 ts_tablet_manager.cc:1398] T e2006432b1c94d4f91cc91b213a28934 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:42.954666 2379 tablet_bootstrap.cc:654] T e2006432b1c94d4f91cc91b213a28934 P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:42.967193 2379 tablet_bootstrap.cc:492] T e2006432b1c94d4f91cc91b213a28934 P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:26:42.967453 2379 ts_tablet_manager.cc:1367] T e2006432b1c94d4f91cc91b213a28934 P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.003s
I20240502 21:26:42.968974 2379 raft_consensus.cc:348] T e2006432b1c94d4f91cc91b213a28934 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:42.969313 2379 raft_consensus.cc:374] T e2006432b1c94d4f91cc91b213a28934 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:42.969461 2379 raft_consensus.cc:729] T e2006432b1c94d4f91cc91b213a28934 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:26:42.969846 2379 consensus_queue.cc:260] T e2006432b1c94d4f91cc91b213a28934 P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
W20240502 21:26:42.971946 1880 tablet.cc:2369] T 815b1fc784714c8cb4ba76707c345625 P 1e04157b98dd4fad98858c290b73c3db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:42.982364 2379 ts_tablet_manager.cc:1398] T e2006432b1c94d4f91cc91b213a28934 P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.015s user 0.007s sys 0.001s
I20240502 21:26:42.986291 31534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815b1fc784714c8cb4ba76707c345625" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:42.986656 31534 raft_consensus.cc:2379] T 815b1fc784714c8cb4ba76707c345625 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e04157b98dd4fad98858c290b73c3db in current term 1: Already voted for candidate 08d7a0680d244b55b93d17f9ab57b84c in this term.
I20240502 21:26:42.988929 2097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90af70ade50047d79664e63328101666" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:42.989205 2097 raft_consensus.cc:2454] T 90af70ade50047d79664e63328101666 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:42.990381 31471 leader_election.cc:302] T 90af70ade50047d79664e63328101666 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:42.990387 1770 leader_election.cc:302] T 815b1fc784714c8cb4ba76707c345625 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db; no voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f
I20240502 21:26:42.990690 1557 raft_consensus.cc:2781] T 90af70ade50047d79664e63328101666 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:42.990777 2359 raft_consensus.cc:3037] T 815b1fc784714c8cb4ba76707c345625 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:42.990803 1557 raft_consensus.cc:482] T 90af70ade50047d79664e63328101666 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:42.990883 1557 raft_consensus.cc:3037] T 90af70ade50047d79664e63328101666 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:42.993265 2359 raft_consensus.cc:2726] T 815b1fc784714c8cb4ba76707c345625 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:26:42.993290 1557 raft_consensus.cc:504] T 90af70ade50047d79664e63328101666 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:42.994065 1557 leader_election.cc:288] T 90af70ade50047d79664e63328101666 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Requested vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:42.995007 2097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90af70ade50047d79664e63328101666" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:26:42.995225 2097 raft_consensus.cc:3037] T 90af70ade50047d79664e63328101666 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:42.998452 2097 raft_consensus.cc:2454] T 90af70ade50047d79664e63328101666 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:43.002570 2231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90af70ade50047d79664e63328101666" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:26:43.002791 2231 raft_consensus.cc:2454] T 90af70ade50047d79664e63328101666 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:43.006942 2231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90af70ade50047d79664e63328101666" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:26:43.007153 2231 raft_consensus.cc:3037] T 90af70ade50047d79664e63328101666 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:43.011112 2231 raft_consensus.cc:2454] T 90af70ade50047d79664e63328101666 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:43.011996 2359 raft_consensus.cc:482] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:43.012146 2359 raft_consensus.cc:504] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:43.012954 2359 leader_election.cc:288] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:26:43.013988 2097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2006432b1c94d4f91cc91b213a28934" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:43.014473 2097 raft_consensus.cc:2454] T e2006432b1c94d4f91cc91b213a28934 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 0.
I20240502 21:26:43.014954 1769 leader_election.cc:302] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 1e04157b98dd4fad98858c290b73c3db; no voters:
I20240502 21:26:43.015262 2359 raft_consensus.cc:2781] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:43.015383 2359 raft_consensus.cc:482] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:43.015465 2359 raft_consensus.cc:3037] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:43.018805 2359 raft_consensus.cc:504] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:43.020226 2359 leader_election.cc:288] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Requested vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:26:43.020331 2096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2006432b1c94d4f91cc91b213a28934" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:26:43.020781 2096 raft_consensus.cc:3037] T e2006432b1c94d4f91cc91b213a28934 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:43.021505 31471 leader_election.cc:302] T 90af70ade50047d79664e63328101666 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:43.021803 1557 raft_consensus.cc:2781] T 90af70ade50047d79664e63328101666 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:43.022025 1557 raft_consensus.cc:686] T 90af70ade50047d79664e63328101666 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:26:43.022266 1557 consensus_queue.cc:237] T 90af70ade50047d79664e63328101666 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:43.023857 2096 raft_consensus.cc:2454] T e2006432b1c94d4f91cc91b213a28934 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:26:43.024344 1769 leader_election.cc:302] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 1e04157b98dd4fad98858c290b73c3db; no voters:
I20240502 21:26:43.025122 2359 raft_consensus.cc:2781] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:43.025327 2359 raft_consensus.cc:686] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Becoming Leader. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Running, Role: LEADER
I20240502 21:26:43.025588 2359 consensus_queue.cc:237] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:43.045061 31392 catalog_manager.cc:5484] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e04157b98dd4fad98858c290b73c3db (127.24.226.133). New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:43.051972 1965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2006432b1c94d4f91cc91b213a28934" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:26:43.052196 1965 raft_consensus.cc:2454] T e2006432b1c94d4f91cc91b213a28934 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 0.
I20240502 21:26:43.052520 1966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2006432b1c94d4f91cc91b213a28934" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf"
I20240502 21:26:43.052678 1966 raft_consensus.cc:3037] T e2006432b1c94d4f91cc91b213a28934 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:43.055826 1966 raft_consensus.cc:2454] T e2006432b1c94d4f91cc91b213a28934 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:26:43.060187 31393 catalog_manager.cc:5484] T 90af70ade50047d79664e63328101666 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 0 to 1, leader changed from <none> to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } }
W20240502 21:26:43.111377 2144 tablet.cc:2369] T 90af70ade50047d79664e63328101666 P 1ade0354c44c4c37ae480c7a94b11e1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:43.130127 31394 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240502 21:26:43.130360 31394 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240502 21:26:43.134467 31394 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 7d7bd0bcc3114425aa0dd1616488d612
I20240502 21:26:43.134910 31394 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 721d6ac1f9294fb4a7280c07bce3bf62
I20240502 21:26:43.135411 31394 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 0057ca44a6c34cecb810133bd0120615
I20240502 21:26:43.135751 31905 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d7bd0bcc3114425aa0dd1616488d612 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:43 UTC) from {username='slave'} at 127.0.0.1:47824
I20240502 21:26:43.136221 31775 tablet_service.cc:1514] Processing DeleteTablet for tablet 721d6ac1f9294fb4a7280c07bce3bf62 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:43 UTC) from {username='slave'} at 127.0.0.1:51596
I20240502 21:26:43.136668 31645 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d7bd0bcc3114425aa0dd1616488d612 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:43 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:43.136945 2328 tablet_replica.cc:330] T 7d7bd0bcc3114425aa0dd1616488d612 P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:26:43.137154 2328 raft_consensus.cc:2229] T 7d7bd0bcc3114425aa0dd1616488d612 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:43.137440 2328 raft_consensus.cc:2258] T 7d7bd0bcc3114425aa0dd1616488d612 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:43.138136 2328 ts_tablet_manager.cc:1875] T 7d7bd0bcc3114425aa0dd1616488d612 P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:43.139019 31515 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d7bd0bcc3114425aa0dd1616488d612 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:43 UTC) from {username='slave'} at 127.0.0.1:45194
I20240502 21:26:43.139274 2329 tablet_replica.cc:330] T 7d7bd0bcc3114425aa0dd1616488d612 P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:43.139510 2329 raft_consensus.cc:2229] T 7d7bd0bcc3114425aa0dd1616488d612 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:43.139823 2329 raft_consensus.cc:2258] T 7d7bd0bcc3114425aa0dd1616488d612 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:43.140524 2329 ts_tablet_manager.cc:1875] T 7d7bd0bcc3114425aa0dd1616488d612 P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:43.140827 2331 tablet_replica.cc:330] T 721d6ac1f9294fb4a7280c07bce3bf62 P 08d7a0680d244b55b93d17f9ab57b84c: stopping tablet replica
I20240502 21:26:43.141041 2331 raft_consensus.cc:2229] T 721d6ac1f9294fb4a7280c07bce3bf62 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:43.141376 2331 raft_consensus.cc:2258] T 721d6ac1f9294fb4a7280c07bce3bf62 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:43.142058 2331 ts_tablet_manager.cc:1875] T 721d6ac1f9294fb4a7280c07bce3bf62 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:43.144596 2330 tablet_replica.cc:330] T 7d7bd0bcc3114425aa0dd1616488d612 P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:43.144904 2330 raft_consensus.cc:2229] T 7d7bd0bcc3114425aa0dd1616488d612 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:43.145234 2330 raft_consensus.cc:2258] T 7d7bd0bcc3114425aa0dd1616488d612 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:43.145591 31906 tablet_service.cc:1514] Processing DeleteTablet for tablet 0057ca44a6c34cecb810133bd0120615 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:43 UTC) from {username='slave'} at 127.0.0.1:47824
I20240502 21:26:43.145926 31905 tablet_service.cc:1514] Processing DeleteTablet for tablet 721d6ac1f9294fb4a7280c07bce3bf62 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:43 UTC) from {username='slave'} at 127.0.0.1:47824
I20240502 21:26:43.146018 2330 ts_tablet_manager.cc:1875] T 7d7bd0bcc3114425aa0dd1616488d612 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:43.146512 31515 tablet_service.cc:1514] Processing DeleteTablet for tablet 721d6ac1f9294fb4a7280c07bce3bf62 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:43 UTC) from {username='slave'} at 127.0.0.1:45194
I20240502 21:26:43.146623 31516 tablet_service.cc:1514] Processing DeleteTablet for tablet 0057ca44a6c34cecb810133bd0120615 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:43 UTC) from {username='slave'} at 127.0.0.1:45194
W20240502 21:26:43.146906 2012 tablet.cc:2369] T e2006432b1c94d4f91cc91b213a28934 P 9dec645b473f40578e726f3e50e219bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:43.147831 2329 ts_tablet_manager.cc:1888] T 7d7bd0bcc3114425aa0dd1616488d612 P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:43.148010 2329 log.cc:1196] T 7d7bd0bcc3114425aa0dd1616488d612 P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/7d7bd0bcc3114425aa0dd1616488d612
I20240502 21:26:43.148465 2329 ts_tablet_manager.cc:1909] T 7d7bd0bcc3114425aa0dd1616488d612 P a44f41b0b6644fb6aa29476aed8c964f: Deleting consensus metadata
I20240502 21:26:43.149363 2329 tablet_replica.cc:330] T 721d6ac1f9294fb4a7280c07bce3bf62 P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:43.149497 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet 7d7bd0bcc3114425aa0dd1616488d612 (table rebalance_test_table_2 [id=ed5c1c20aa174a8e845358f36f4420a9]) successfully deleted
I20240502 21:26:43.149538 2329 raft_consensus.cc:2229] T 721d6ac1f9294fb4a7280c07bce3bf62 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:43.149768 2329 raft_consensus.cc:2258] T 721d6ac1f9294fb4a7280c07bce3bf62 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:43.152587 31645 tablet_service.cc:1514] Processing DeleteTablet for tablet 0057ca44a6c34cecb810133bd0120615 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:43 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:43.159976 2329 ts_tablet_manager.cc:1875] T 721d6ac1f9294fb4a7280c07bce3bf62 P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:43.160019 31775 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 21:26:43.166836 1946 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:43.170656 31645 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 21:26:43.171015 2210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:43.174993 2329 ts_tablet_manager.cc:1888] T 721d6ac1f9294fb4a7280c07bce3bf62 P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:43.175120 2329 log.cc:1196] T 721d6ac1f9294fb4a7280c07bce3bf62 P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/721d6ac1f9294fb4a7280c07bce3bf62
I20240502 21:26:43.175755 2329 ts_tablet_manager.cc:1909] T 721d6ac1f9294fb4a7280c07bce3bf62 P a44f41b0b6644fb6aa29476aed8c964f: Deleting consensus metadata
I20240502 21:26:43.176602 2329 tablet_replica.cc:330] T 0057ca44a6c34cecb810133bd0120615 P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:43.176821 2329 raft_consensus.cc:2229] T 0057ca44a6c34cecb810133bd0120615 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:43.177320 2329 raft_consensus.cc:2258] T 0057ca44a6c34cecb810133bd0120615 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:43.177843 31516 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:26:43.178004 2329 ts_tablet_manager.cc:1875] T 0057ca44a6c34cecb810133bd0120615 P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:43.178719 31394 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:26:43.180091 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet 721d6ac1f9294fb4a7280c07bce3bf62 (table rebalance_test_table_2 [id=ed5c1c20aa174a8e845358f36f4420a9]) successfully deleted
I20240502 21:26:43.184039 2077 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:43.186211 2329 ts_tablet_manager.cc:1888] T 0057ca44a6c34cecb810133bd0120615 P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:43.186393 2329 log.cc:1196] T 0057ca44a6c34cecb810133bd0120615 P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/0057ca44a6c34cecb810133bd0120615
I20240502 21:26:43.186807 2329 ts_tablet_manager.cc:1909] T 0057ca44a6c34cecb810133bd0120615 P a44f41b0b6644fb6aa29476aed8c964f: Deleting consensus metadata
I20240502 21:26:43.191918 2328 ts_tablet_manager.cc:1888] T 7d7bd0bcc3114425aa0dd1616488d612 P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:43.192036 2328 log.cc:1196] T 7d7bd0bcc3114425aa0dd1616488d612 P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/7d7bd0bcc3114425aa0dd1616488d612
I20240502 21:26:43.192420 2328 ts_tablet_manager.cc:1909] T 7d7bd0bcc3114425aa0dd1616488d612 P 2d50f33557944bb480ea32199d9ee4f1: Deleting consensus metadata
I20240502 21:26:43.193519 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet 0057ca44a6c34cecb810133bd0120615 (table rebalance_test_table_2 [id=ed5c1c20aa174a8e845358f36f4420a9]) successfully deleted
I20240502 21:26:43.194092 1814 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:43.195525 2330 ts_tablet_manager.cc:1888] T 7d7bd0bcc3114425aa0dd1616488d612 P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:43.195676 2330 log.cc:1196] T 7d7bd0bcc3114425aa0dd1616488d612 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/7d7bd0bcc3114425aa0dd1616488d612
I20240502 21:26:43.196040 2330 ts_tablet_manager.cc:1909] T 7d7bd0bcc3114425aa0dd1616488d612 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:26:43.196943 2330 tablet_replica.cc:330] T 0057ca44a6c34cecb810133bd0120615 P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:43.197026 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet 7d7bd0bcc3114425aa0dd1616488d612 (table rebalance_test_table_2 [id=ed5c1c20aa174a8e845358f36f4420a9]) successfully deleted
I20240502 21:26:43.197137 2330 raft_consensus.cc:2229] T 0057ca44a6c34cecb810133bd0120615 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:43.197438 2330 raft_consensus.cc:2258] T 0057ca44a6c34cecb810133bd0120615 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:43.198063 2330 ts_tablet_manager.cc:1875] T 0057ca44a6c34cecb810133bd0120615 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:43.198663 2331 ts_tablet_manager.cc:1888] T 721d6ac1f9294fb4a7280c07bce3bf62 P 08d7a0680d244b55b93d17f9ab57b84c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:43.198828 2331 log.cc:1196] T 721d6ac1f9294fb4a7280c07bce3bf62 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/wals/721d6ac1f9294fb4a7280c07bce3bf62
I20240502 21:26:43.199445 2331 ts_tablet_manager.cc:1909] T 721d6ac1f9294fb4a7280c07bce3bf62 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting consensus metadata
I20240502 21:26:43.200573 31381 catalog_manager.cc:4830] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): tablet 721d6ac1f9294fb4a7280c07bce3bf62 (table rebalance_test_table_2 [id=ed5c1c20aa174a8e845358f36f4420a9]) successfully deleted
I20240502 21:26:43.207036 2330 ts_tablet_manager.cc:1888] T 0057ca44a6c34cecb810133bd0120615 P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:43.207209 2330 log.cc:1196] T 0057ca44a6c34cecb810133bd0120615 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/0057ca44a6c34cecb810133bd0120615
I20240502 21:26:43.207734 2330 ts_tablet_manager.cc:1909] T 0057ca44a6c34cecb810133bd0120615 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:26:43.208920 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet 0057ca44a6c34cecb810133bd0120615 (table rebalance_test_table_2 [id=ed5c1c20aa174a8e845358f36f4420a9]) successfully deleted
I20240502 21:26:43.213371 2328 tablet_replica.cc:330] T 0057ca44a6c34cecb810133bd0120615 P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:26:43.213831 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet 7d7bd0bcc3114425aa0dd1616488d612 (table rebalance_test_table_2 [id=ed5c1c20aa174a8e845358f36f4420a9]) successfully deleted
I20240502 21:26:43.214887 2328 raft_consensus.cc:2229] T 0057ca44a6c34cecb810133bd0120615 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:43.217350 2328 raft_consensus.cc:2258] T 0057ca44a6c34cecb810133bd0120615 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:43.218024 2328 ts_tablet_manager.cc:1875] T 0057ca44a6c34cecb810133bd0120615 P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:43.220628 31905 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 21:26:43.235776 31863 consensus_peers.cc:485] T 721d6ac1f9294fb4a7280c07bce3bf62 P 2d50f33557944bb480ea32199d9ee4f1 -> Peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Couldn't send request to peer a44f41b0b6644fb6aa29476aed8c964f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 721d6ac1f9294fb4a7280c07bce3bf62. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:43.236835 2328 ts_tablet_manager.cc:1888] T 0057ca44a6c34cecb810133bd0120615 P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:43.236980 2328 log.cc:1196] T 0057ca44a6c34cecb810133bd0120615 P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/0057ca44a6c34cecb810133bd0120615
I20240502 21:26:43.238734 2328 ts_tablet_manager.cc:1909] T 0057ca44a6c34cecb810133bd0120615 P 2d50f33557944bb480ea32199d9ee4f1: Deleting consensus metadata
I20240502 21:26:43.239696 2328 tablet_replica.cc:330] T 721d6ac1f9294fb4a7280c07bce3bf62 P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:26:43.239854 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet 0057ca44a6c34cecb810133bd0120615 (table rebalance_test_table_2 [id=ed5c1c20aa174a8e845358f36f4420a9]) successfully deleted
I20240502 21:26:43.242739 2328 raft_consensus.cc:2229] T 721d6ac1f9294fb4a7280c07bce3bf62 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:43.243260 2328 raft_consensus.cc:2258] T 721d6ac1f9294fb4a7280c07bce3bf62 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:43.244109 2328 ts_tablet_manager.cc:1875] T 721d6ac1f9294fb4a7280c07bce3bf62 P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:43.260460 2328 ts_tablet_manager.cc:1888] T 721d6ac1f9294fb4a7280c07bce3bf62 P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:43.261744 2328 log.cc:1196] T 721d6ac1f9294fb4a7280c07bce3bf62 P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/721d6ac1f9294fb4a7280c07bce3bf62
I20240502 21:26:43.262182 2328 ts_tablet_manager.cc:1909] T 721d6ac1f9294fb4a7280c07bce3bf62 P 2d50f33557944bb480ea32199d9ee4f1: Deleting consensus metadata
I20240502 21:26:43.263060 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet 721d6ac1f9294fb4a7280c07bce3bf62 (table rebalance_test_table_2 [id=ed5c1c20aa174a8e845358f36f4420a9]) successfully deleted
I20240502 21:26:43.269912 1966 raft_consensus.cc:1261] T e2006432b1c94d4f91cc91b213a28934 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:43.270563 2359 consensus_queue.cc:1035] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:43.280355 2096 raft_consensus.cc:1261] T e2006432b1c94d4f91cc91b213a28934 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:43.281324 2359 consensus_queue.cc:1035] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:43.310268 31394 catalog_manager.cc:5576] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753) does not have the latest schema for tablet e2006432b1c94d4f91cc91b213a28934 (table rebalancer_dynamic_table [id=bc87b3ebaa8c4f1b858fb14c03cb7504]). Expected version 1 got 0
I20240502 21:26:43.310596 2412 tablet.cc:1722] T e2006432b1c94d4f91cc91b213a28934 P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:43.310995 1690 consensus_queue.cc:1035] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:43.311273 2412 tablet.cc:1620] T e2006432b1c94d4f91cc91b213a28934 P 9dec645b473f40578e726f3e50e219bf: MemRowSet was empty: no flush needed.
I20240502 21:26:43.313225 2406 mvcc.cc:205] Tried to move back new op lower bound from 7023350592553963520 to 7023350591598505984. Current Snapshot: MvccSnapshot[applied={T|T < 7023350592553963520}]
I20240502 21:26:43.319382 31394 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=bc87b3ebaa8c4f1b858fb14c03cb7504] alter complete (version 1)
I20240502 21:26:43.336896 31795 raft_consensus.cc:3037] T b9342dbe0497499ba4c238f2ed033293 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:43.339525 2411 tablet.cc:1722] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:43.340282 2356 consensus_queue.cc:1035] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:43.353942 2411 tablet.cc:1620] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db: MemRowSet was empty: no flush needed.
I20240502 21:26:43.357213 2370 consensus_queue.cc:1035] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:43.366605 2357 consensus_queue.cc:1035] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:43.394599 2413 tablet.cc:1722] T e2006432b1c94d4f91cc91b213a28934 P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:43.396420 31392 catalog_manager.cc:5576] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569) does not have the latest schema for tablet e2006432b1c94d4f91cc91b213a28934 (table rebalancer_dynamic_table [id=bc87b3ebaa8c4f1b858fb14c03cb7504]). Expected version 1 got 0
I20240502 21:26:43.398845 2413 tablet.cc:1620] T e2006432b1c94d4f91cc91b213a28934 P 1ade0354c44c4c37ae480c7a94b11e1b: MemRowSet was empty: no flush needed.
I20240502 21:26:43.417194 2303 consensus_queue.cc:1035] T 90af70ade50047d79664e63328101666 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:43.422891 2303 consensus_queue.cc:1035] T 90af70ade50047d79664e63328101666 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:43.430130 31394 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:26:43.439384 2411 tablet.cc:1722] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:43.439838 2411 tablet.cc:1620] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db: MemRowSet was empty: no flush needed.
I20240502 21:26:43.440783 2413 tablet.cc:1722] T e2006432b1c94d4f91cc91b213a28934 P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:43.441313 2413 tablet.cc:1620] T e2006432b1c94d4f91cc91b213a28934 P 1ade0354c44c4c37ae480c7a94b11e1b: MemRowSet was empty: no flush needed.
I20240502 21:26:43.441829 2412 tablet.cc:1722] T e2006432b1c94d4f91cc91b213a28934 P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:43.442351 2412 tablet.cc:1620] T e2006432b1c94d4f91cc91b213a28934 P 9dec645b473f40578e726f3e50e219bf: MemRowSet was empty: no flush needed.
I20240502 21:26:43.454360 31394 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=bc87b3ebaa8c4f1b858fb14c03cb7504] alter complete (version 2)
I20240502 21:26:43.559918 31391 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:26:43.574093 2411 tablet.cc:1722] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:43.581243 2413 tablet.cc:1722] T e2006432b1c94d4f91cc91b213a28934 P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:43.582290 2412 tablet.cc:1722] T e2006432b1c94d4f91cc91b213a28934 P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:43.593930 31394 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table_new_name [id=bc87b3ebaa8c4f1b858fb14c03cb7504] alter complete (version 3)
I20240502 21:26:43.647132 31390 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240502 21:26:43.647464 31390 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240502 21:26:43.654670 31394 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_extra_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:43.657505 31390 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet ce9c58ac50a74f8e871170936e967d64
I20240502 21:26:43.658205 31516 tablet_service.cc:1514] Processing DeleteTablet for tablet ce9c58ac50a74f8e871170936e967d64 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:43 UTC) from {username='slave'} at 127.0.0.1:45194
I20240502 21:26:43.658308 31775 tablet_service.cc:1514] Processing DeleteTablet for tablet ce9c58ac50a74f8e871170936e967d64 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:43 UTC) from {username='slave'} at 127.0.0.1:51596
I20240502 21:26:43.658515 2329 tablet_replica.cc:330] T ce9c58ac50a74f8e871170936e967d64 P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:43.658721 2329 raft_consensus.cc:2229] T ce9c58ac50a74f8e871170936e967d64 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:43.658835 31390 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 69faa069cb6e4a86811ce8de61b34f90
I20240502 21:26:43.659036 2329 raft_consensus.cc:2258] T ce9c58ac50a74f8e871170936e967d64 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:43.659204 31390 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 0645323dfbad46ceb1e11898968e9d22
I20240502 21:26:43.659710 2329 ts_tablet_manager.cc:1875] T ce9c58ac50a74f8e871170936e967d64 P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:43.659745 31775 tablet_service.cc:1514] Processing DeleteTablet for tablet 0645323dfbad46ceb1e11898968e9d22 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:43 UTC) from {username='slave'} at 127.0.0.1:51596
I20240502 21:26:43.661659 2331 tablet_replica.cc:330] T ce9c58ac50a74f8e871170936e967d64 P 08d7a0680d244b55b93d17f9ab57b84c: stopping tablet replica
I20240502 21:26:43.661860 2331 raft_consensus.cc:2229] T ce9c58ac50a74f8e871170936e967d64 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:43.662060 2331 raft_consensus.cc:2258] T ce9c58ac50a74f8e871170936e967d64 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:43.662626 2331 ts_tablet_manager.cc:1875] T ce9c58ac50a74f8e871170936e967d64 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:43.665761 31905 tablet_service.cc:1514] Processing DeleteTablet for tablet 69faa069cb6e4a86811ce8de61b34f90 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:43 UTC) from {username='slave'} at 127.0.0.1:47824
I20240502 21:26:43.665974 31906 tablet_service.cc:1514] Processing DeleteTablet for tablet 0645323dfbad46ceb1e11898968e9d22 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:43 UTC) from {username='slave'} at 127.0.0.1:47824
I20240502 21:26:43.666028 2328 tablet_replica.cc:330] T 69faa069cb6e4a86811ce8de61b34f90 P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:26:43.666205 2328 raft_consensus.cc:2229] T 69faa069cb6e4a86811ce8de61b34f90 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:43.666430 2328 raft_consensus.cc:2258] T 69faa069cb6e4a86811ce8de61b34f90 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:43.666529 31645 tablet_service.cc:1514] Processing DeleteTablet for tablet ce9c58ac50a74f8e871170936e967d64 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:43 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:43.666817 31647 tablet_service.cc:1514] Processing DeleteTablet for tablet 69faa069cb6e4a86811ce8de61b34f90 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:43 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:43.667062 2328 ts_tablet_manager.cc:1875] T 69faa069cb6e4a86811ce8de61b34f90 P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:43.667438 2330 tablet_replica.cc:330] T ce9c58ac50a74f8e871170936e967d64 P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:43.667461 31516 tablet_service.cc:1514] Processing DeleteTablet for tablet 69faa069cb6e4a86811ce8de61b34f90 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:43 UTC) from {username='slave'} at 127.0.0.1:45194
I20240502 21:26:43.667650 2330 raft_consensus.cc:2229] T ce9c58ac50a74f8e871170936e967d64 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:43.667845 2330 raft_consensus.cc:2258] T ce9c58ac50a74f8e871170936e967d64 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:43.668476 2330 ts_tablet_manager.cc:1875] T ce9c58ac50a74f8e871170936e967d64 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:43.668614 31515 tablet_service.cc:1514] Processing DeleteTablet for tablet 0645323dfbad46ceb1e11898968e9d22 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:43 UTC) from {username='slave'} at 127.0.0.1:45194
I20240502 21:26:43.671442 1813 tablet_service.cc:1467] Processing CreateTablet for tablet f74ea10ab8a2459794f816707b87f494 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=408b4cadaf474250a5666b6a49f4f8cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:43.671870 1813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f74ea10ab8a2459794f816707b87f494. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:43.674477 2328 ts_tablet_manager.cc:1888] T 69faa069cb6e4a86811ce8de61b34f90 P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:43.674607 2328 log.cc:1196] T 69faa069cb6e4a86811ce8de61b34f90 P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/69faa069cb6e4a86811ce8de61b34f90
I20240502 21:26:43.675031 2328 ts_tablet_manager.cc:1909] T 69faa069cb6e4a86811ce8de61b34f90 P 2d50f33557944bb480ea32199d9ee4f1: Deleting consensus metadata
I20240502 21:26:43.676010 2328 tablet_replica.cc:330] T 0645323dfbad46ceb1e11898968e9d22 P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:26:43.676131 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet 69faa069cb6e4a86811ce8de61b34f90 (table rebalance_test_table_4 [id=223f9daed2194c19b553fe65892428cf]) successfully deleted
I20240502 21:26:43.676417 2328 raft_consensus.cc:2229] T 0645323dfbad46ceb1e11898968e9d22 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:43.676805 2328 raft_consensus.cc:2258] T 0645323dfbad46ceb1e11898968e9d22 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:43.676975 31647 tablet_service.cc:1467] Processing CreateTablet for tablet 49acc5301b9849e9b6795179acbc59a8 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=408b4cadaf474250a5666b6a49f4f8cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:43.677306 31647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49acc5301b9849e9b6795179acbc59a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:43.677750 2328 ts_tablet_manager.cc:1875] T 0645323dfbad46ceb1e11898968e9d22 P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:43.677533 1946 tablet_service.cc:1467] Processing CreateTablet for tablet f02f83ec0a494da5b66241d9b379a03f (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=408b4cadaf474250a5666b6a49f4f8cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:43.678117 1946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f02f83ec0a494da5b66241d9b379a03f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:43.680804 2329 ts_tablet_manager.cc:1888] T ce9c58ac50a74f8e871170936e967d64 P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:43.680897 2329 log.cc:1196] T ce9c58ac50a74f8e871170936e967d64 P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/ce9c58ac50a74f8e871170936e967d64
I20240502 21:26:43.681314 2329 ts_tablet_manager.cc:1909] T ce9c58ac50a74f8e871170936e967d64 P a44f41b0b6644fb6aa29476aed8c964f: Deleting consensus metadata
I20240502 21:26:43.682174 2329 tablet_replica.cc:330] T 69faa069cb6e4a86811ce8de61b34f90 P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:43.682363 2329 raft_consensus.cc:2229] T 69faa069cb6e4a86811ce8de61b34f90 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:43.682556 2329 raft_consensus.cc:2258] T 69faa069cb6e4a86811ce8de61b34f90 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:43.683346 2209 tablet_service.cc:1467] Processing CreateTablet for tablet 49acc5301b9849e9b6795179acbc59a8 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=408b4cadaf474250a5666b6a49f4f8cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:43.683646 2077 tablet_service.cc:1467] Processing CreateTablet for tablet f74ea10ab8a2459794f816707b87f494 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=408b4cadaf474250a5666b6a49f4f8cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:43.683926 2209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49acc5301b9849e9b6795179acbc59a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:43.684062 2077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f74ea10ab8a2459794f816707b87f494. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:43.684391 2328 ts_tablet_manager.cc:1888] T 0645323dfbad46ceb1e11898968e9d22 P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:43.684509 2328 log.cc:1196] T 0645323dfbad46ceb1e11898968e9d22 P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/0645323dfbad46ceb1e11898968e9d22
I20240502 21:26:43.684952 2328 ts_tablet_manager.cc:1909] T 0645323dfbad46ceb1e11898968e9d22 P 2d50f33557944bb480ea32199d9ee4f1: Deleting consensus metadata
I20240502 21:26:43.688227 2076 tablet_service.cc:1467] Processing CreateTablet for tablet 49acc5301b9849e9b6795179acbc59a8 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=408b4cadaf474250a5666b6a49f4f8cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:43.688300 2466 tablet_bootstrap.cc:492] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:43.688594 2076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49acc5301b9849e9b6795179acbc59a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:43.690106 2466 tablet_bootstrap.cc:654] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:43.690915 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet 0645323dfbad46ceb1e11898968e9d22 (table rebalance_test_table_4 [id=223f9daed2194c19b553fe65892428cf]) successfully deleted
I20240502 21:26:43.691608 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet ce9c58ac50a74f8e871170936e967d64 (table rebalance_test_table_4 [id=223f9daed2194c19b553fe65892428cf]) successfully deleted
I20240502 21:26:43.692215 2466 tablet_bootstrap.cc:492] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:26:43.692258 2331 ts_tablet_manager.cc:1888] T ce9c58ac50a74f8e871170936e967d64 P 08d7a0680d244b55b93d17f9ab57b84c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:43.692353 2466 ts_tablet_manager.cc:1367] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:26:43.692379 2331 log.cc:1196] T ce9c58ac50a74f8e871170936e967d64 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/wals/ce9c58ac50a74f8e871170936e967d64
I20240502 21:26:43.692795 2331 ts_tablet_manager.cc:1909] T ce9c58ac50a74f8e871170936e967d64 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting consensus metadata
I20240502 21:26:43.692788 2075 tablet_service.cc:1467] Processing CreateTablet for tablet f02f83ec0a494da5b66241d9b379a03f (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=408b4cadaf474250a5666b6a49f4f8cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:43.693176 2075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f02f83ec0a494da5b66241d9b379a03f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:43.693403 2466 raft_consensus.cc:348] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:43.693585 2466 raft_consensus.cc:374] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:43.693684 2466 raft_consensus.cc:729] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:26:43.694002 2466 consensus_queue.cc:260] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:43.694630 2466 ts_tablet_manager.cc:1398] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:43.696130 2331 tablet_replica.cc:330] T 0645323dfbad46ceb1e11898968e9d22 P 08d7a0680d244b55b93d17f9ab57b84c: stopping tablet replica
I20240502 21:26:43.696333 2331 raft_consensus.cc:2229] T 0645323dfbad46ceb1e11898968e9d22 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:43.696575 2331 raft_consensus.cc:2258] T 0645323dfbad46ceb1e11898968e9d22 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:43.696949 2208 tablet_service.cc:1467] Processing CreateTablet for tablet f02f83ec0a494da5b66241d9b379a03f (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=408b4cadaf474250a5666b6a49f4f8cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:43.697171 2331 ts_tablet_manager.cc:1875] T 0645323dfbad46ceb1e11898968e9d22 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:43.697322 2208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f02f83ec0a494da5b66241d9b379a03f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:43.698856 31381 catalog_manager.cc:4830] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): tablet ce9c58ac50a74f8e871170936e967d64 (table rebalance_test_table_4 [id=223f9daed2194c19b553fe65892428cf]) successfully deleted
I20240502 21:26:43.701745 31390 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:43.701905 31390 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:43.702701 2329 ts_tablet_manager.cc:1875] T 69faa069cb6e4a86811ce8de61b34f90 P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:43.703894 2330 ts_tablet_manager.cc:1888] T ce9c58ac50a74f8e871170936e967d64 P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:43.713703 2210 tablet_service.cc:1467] Processing CreateTablet for tablet f74ea10ab8a2459794f816707b87f494 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=408b4cadaf474250a5666b6a49f4f8cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:43.714120 2210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f74ea10ab8a2459794f816707b87f494. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:43.714968 2330 log.cc:1196] T ce9c58ac50a74f8e871170936e967d64 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/ce9c58ac50a74f8e871170936e967d64
I20240502 21:26:43.717136 2330 ts_tablet_manager.cc:1909] T ce9c58ac50a74f8e871170936e967d64 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:26:43.719244 31390 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet e2006432b1c94d4f91cc91b213a28934
I20240502 21:26:43.719592 2330 tablet_replica.cc:330] T 69faa069cb6e4a86811ce8de61b34f90 P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:43.719838 2330 raft_consensus.cc:2229] T 69faa069cb6e4a86811ce8de61b34f90 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:43.720001 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet ce9c58ac50a74f8e871170936e967d64 (table rebalance_test_table_4 [id=223f9daed2194c19b553fe65892428cf]) successfully deleted
I20240502 21:26:43.720691 1946 tablet_service.cc:1514] Processing DeleteTablet for tablet e2006432b1c94d4f91cc91b213a28934 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:43 UTC) from {username='slave'} at 127.0.0.1:33002
I20240502 21:26:43.721004 1814 tablet_service.cc:1514] Processing DeleteTablet for tablet e2006432b1c94d4f91cc91b213a28934 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:43 UTC) from {username='slave'} at 127.0.0.1:47510
I20240502 21:26:43.729840 2076 tablet_service.cc:1514] Processing DeleteTablet for tablet e2006432b1c94d4f91cc91b213a28934 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:43 UTC) from {username='slave'} at 127.0.0.1:36740
I20240502 21:26:43.730613 2468 tablet_bootstrap.cc:492] T 49acc5301b9849e9b6795179acbc59a8 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:43.720192 2330 raft_consensus.cc:2258] T 69faa069cb6e4a86811ce8de61b34f90 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:43.732087 2330 ts_tablet_manager.cc:1875] T 69faa069cb6e4a86811ce8de61b34f90 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:43.732270 2468 tablet_bootstrap.cc:654] T 49acc5301b9849e9b6795179acbc59a8 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:43.733433 2329 ts_tablet_manager.cc:1888] T 69faa069cb6e4a86811ce8de61b34f90 P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:43.733609 2329 log.cc:1196] T 69faa069cb6e4a86811ce8de61b34f90 P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/69faa069cb6e4a86811ce8de61b34f90
I20240502 21:26:43.734051 2329 ts_tablet_manager.cc:1909] T 69faa069cb6e4a86811ce8de61b34f90 P a44f41b0b6644fb6aa29476aed8c964f: Deleting consensus metadata
I20240502 21:26:43.734949 2329 tablet_replica.cc:330] T 0645323dfbad46ceb1e11898968e9d22 P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:43.735922 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet 69faa069cb6e4a86811ce8de61b34f90 (table rebalance_test_table_4 [id=223f9daed2194c19b553fe65892428cf]) successfully deleted
I20240502 21:26:43.739125 2470 tablet_bootstrap.cc:492] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:43.741273 2470 tablet_bootstrap.cc:654] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:43.743481 2478 tablet_replica.cc:330] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db: stopping tablet replica
I20240502 21:26:43.743988 2478 raft_consensus.cc:2229] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:43.744457 2331 ts_tablet_manager.cc:1888] T 0645323dfbad46ceb1e11898968e9d22 P 08d7a0680d244b55b93d17f9ab57b84c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:43.745033 2331 log.cc:1196] T 0645323dfbad46ceb1e11898968e9d22 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/wals/0645323dfbad46ceb1e11898968e9d22
I20240502 21:26:43.745779 2331 ts_tablet_manager.cc:1909] T 0645323dfbad46ceb1e11898968e9d22 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting consensus metadata
W20240502 21:26:43.752928 31471 consensus_peers.cc:485] T 0645323dfbad46ceb1e11898968e9d22 P a44f41b0b6644fb6aa29476aed8c964f -> Peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): Couldn't send request to peer 08d7a0680d244b55b93d17f9ab57b84c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:43.743882 2479 tablet_replica.cc:330] T e2006432b1c94d4f91cc91b213a28934 P 1ade0354c44c4c37ae480c7a94b11e1b: stopping tablet replica
I20240502 21:26:43.754216 2479 raft_consensus.cc:2229] T e2006432b1c94d4f91cc91b213a28934 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:43.754704 2479 raft_consensus.cc:2258] T e2006432b1c94d4f91cc91b213a28934 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:43.757413 31381 catalog_manager.cc:4830] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): tablet 0645323dfbad46ceb1e11898968e9d22 (table rebalance_test_table_4 [id=223f9daed2194c19b553fe65892428cf]) successfully deleted
I20240502 21:26:43.758891 2329 raft_consensus.cc:2229] T 0645323dfbad46ceb1e11898968e9d22 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:43.759702 2329 raft_consensus.cc:2258] T 0645323dfbad46ceb1e11898968e9d22 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:43.760581 2479 ts_tablet_manager.cc:1875] T e2006432b1c94d4f91cc91b213a28934 P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:43.762970 2329 ts_tablet_manager.cc:1875] T 0645323dfbad46ceb1e11898968e9d22 P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:43.769064 2468 tablet_bootstrap.cc:492] T 49acc5301b9849e9b6795179acbc59a8 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:43.769243 2468 ts_tablet_manager.cc:1367] T 49acc5301b9849e9b6795179acbc59a8 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.039s user 0.011s sys 0.014s
I20240502 21:26:43.770166 2468 raft_consensus.cc:348] T 49acc5301b9849e9b6795179acbc59a8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:43.770409 2468 raft_consensus.cc:374] T 49acc5301b9849e9b6795179acbc59a8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:43.770507 2468 raft_consensus.cc:729] T 49acc5301b9849e9b6795179acbc59a8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:43.773540 2330 ts_tablet_manager.cc:1888] T 69faa069cb6e4a86811ce8de61b34f90 P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:43.773407 2468 consensus_queue.cc:260] T 49acc5301b9849e9b6795179acbc59a8 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:43.773702 2330 log.cc:1196] T 69faa069cb6e4a86811ce8de61b34f90 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/69faa069cb6e4a86811ce8de61b34f90
I20240502 21:26:43.774230 2330 ts_tablet_manager.cc:1909] T 69faa069cb6e4a86811ce8de61b34f90 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:26:43.775715 2477 tablet_replica.cc:330] T e2006432b1c94d4f91cc91b213a28934 P 9dec645b473f40578e726f3e50e219bf: stopping tablet replica
I20240502 21:26:43.775715 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet 69faa069cb6e4a86811ce8de61b34f90 (table rebalance_test_table_4 [id=223f9daed2194c19b553fe65892428cf]) successfully deleted
I20240502 21:26:43.775985 2477 raft_consensus.cc:2229] T e2006432b1c94d4f91cc91b213a28934 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:43.776268 2477 raft_consensus.cc:2258] T e2006432b1c94d4f91cc91b213a28934 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:43.777271 2477 ts_tablet_manager.cc:1875] T e2006432b1c94d4f91cc91b213a28934 P 9dec645b473f40578e726f3e50e219bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:43.777297 2468 ts_tablet_manager.cc:1398] T 49acc5301b9849e9b6795179acbc59a8 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240502 21:26:43.780481 2468 tablet_bootstrap.cc:492] T f02f83ec0a494da5b66241d9b379a03f P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:43.782454 2468 tablet_bootstrap.cc:654] T f02f83ec0a494da5b66241d9b379a03f P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:43.788836 2478 raft_consensus.cc:2258] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:43.790377 2472 tablet_bootstrap.cc:492] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:43.792625 2472 tablet_bootstrap.cc:654] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:43.793108 2474 tablet_bootstrap.cc:492] T f74ea10ab8a2459794f816707b87f494 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:43.795326 2474 tablet_bootstrap.cc:654] T f74ea10ab8a2459794f816707b87f494 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:43.796531 2478 ts_tablet_manager.cc:1875] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:43.805534 2478 ts_tablet_manager.cc:1888] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:43.805755 2478 log.cc:1196] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal/wals/e2006432b1c94d4f91cc91b213a28934
I20240502 21:26:43.806295 2478 ts_tablet_manager.cc:1909] T e2006432b1c94d4f91cc91b213a28934 P 1e04157b98dd4fad98858c290b73c3db: Deleting consensus metadata
I20240502 21:26:43.762092 2470 tablet_bootstrap.cc:492] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:43.813556 2470 ts_tablet_manager.cc:1367] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.075s user 0.000s sys 0.005s
I20240502 21:26:43.818990 31775 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 21:26:43.822340 2470 raft_consensus.cc:348] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:43.822566 2470 raft_consensus.cc:374] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:43.822706 2470 raft_consensus.cc:729] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:43.822947 2470 consensus_queue.cc:260] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:43.823650 2470 ts_tablet_manager.cc:1398] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240502 21:26:43.825821 31390 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:26:43.831854 2470 tablet_bootstrap.cc:492] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:43.834146 2470 tablet_bootstrap.cc:654] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:43.835721 1946 tablet_service.cc:1467] Processing CreateTablet for tablet 530f72c556e84bdbbf78fd5d29b6e4db (DEFAULT_TABLE table=rebalancer_dynamic_table [id=acb3e4c884724fdab17608b6d200b2e3]), partition=
I20240502 21:26:43.835916 2474 tablet_bootstrap.cc:492] T f74ea10ab8a2459794f816707b87f494 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:43.836086 1946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 530f72c556e84bdbbf78fd5d29b6e4db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:43.836076 2474 ts_tablet_manager.cc:1367] T f74ea10ab8a2459794f816707b87f494 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.043s user 0.000s sys 0.003s
I20240502 21:26:43.837038 2474 raft_consensus.cc:348] T f74ea10ab8a2459794f816707b87f494 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:43.837229 2474 raft_consensus.cc:374] T f74ea10ab8a2459794f816707b87f494 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:43.837313 2474 raft_consensus.cc:729] T f74ea10ab8a2459794f816707b87f494 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:43.837524 2474 consensus_queue.cc:260] T f74ea10ab8a2459794f816707b87f494 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:43.840760 2209 tablet_service.cc:1467] Processing CreateTablet for tablet 530f72c556e84bdbbf78fd5d29b6e4db (DEFAULT_TABLE table=rebalancer_dynamic_table [id=acb3e4c884724fdab17608b6d200b2e3]), partition=
I20240502 21:26:43.841109 2209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 530f72c556e84bdbbf78fd5d29b6e4db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:43.842558 2329 ts_tablet_manager.cc:1888] T 0645323dfbad46ceb1e11898968e9d22 P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:43.842687 2329 log.cc:1196] T 0645323dfbad46ceb1e11898968e9d22 P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/0645323dfbad46ceb1e11898968e9d22
I20240502 21:26:43.842794 2466 tablet_bootstrap.cc:492] T 530f72c556e84bdbbf78fd5d29b6e4db P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:43.843133 2329 ts_tablet_manager.cc:1909] T 0645323dfbad46ceb1e11898968e9d22 P a44f41b0b6644fb6aa29476aed8c964f: Deleting consensus metadata
I20240502 21:26:43.844416 2466 tablet_bootstrap.cc:654] T 530f72c556e84bdbbf78fd5d29b6e4db P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:43.845541 2477 ts_tablet_manager.cc:1888] T e2006432b1c94d4f91cc91b213a28934 P 9dec645b473f40578e726f3e50e219bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:43.845669 2477 log.cc:1196] T e2006432b1c94d4f91cc91b213a28934 P 9dec645b473f40578e726f3e50e219bf: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal/wals/e2006432b1c94d4f91cc91b213a28934
I20240502 21:26:43.846089 2477 ts_tablet_manager.cc:1909] T e2006432b1c94d4f91cc91b213a28934 P 9dec645b473f40578e726f3e50e219bf: Deleting consensus metadata
I20240502 21:26:43.852095 2474 ts_tablet_manager.cc:1398] T f74ea10ab8a2459794f816707b87f494 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.016s user 0.000s sys 0.002s
I20240502 21:26:43.852308 1946 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:43.862463 2472 tablet_bootstrap.cc:492] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:43.862715 2472 ts_tablet_manager.cc:1367] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.072s user 0.004s sys 0.000s
I20240502 21:26:43.863459 2472 raft_consensus.cc:348] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:43.863626 2472 raft_consensus.cc:374] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:43.863756 2472 raft_consensus.cc:729] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:43.863977 2472 consensus_queue.cc:260] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:43.864617 2472 ts_tablet_manager.cc:1398] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:43.865753 1813 tablet_service.cc:1467] Processing CreateTablet for tablet 530f72c556e84bdbbf78fd5d29b6e4db (DEFAULT_TABLE table=rebalancer_dynamic_table [id=acb3e4c884724fdab17608b6d200b2e3]), partition=
I20240502 21:26:43.865942 2470 tablet_bootstrap.cc:492] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:43.866078 1813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 530f72c556e84bdbbf78fd5d29b6e4db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:43.866094 2470 ts_tablet_manager.cc:1367] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.042s user 0.000s sys 0.004s
I20240502 21:26:43.866766 2470 raft_consensus.cc:348] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:43.866953 2470 raft_consensus.cc:374] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:43.867007 2470 raft_consensus.cc:729] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:43.868857 31379 catalog_manager.cc:4830] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): tablet e2006432b1c94d4f91cc91b213a28934 (table rebalancer_dynamic_table_new_name [id=bc87b3ebaa8c4f1b858fb14c03cb7504]) successfully deleted
I20240502 21:26:43.869588 31381 catalog_manager.cc:4830] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): tablet e2006432b1c94d4f91cc91b213a28934 (table rebalancer_dynamic_table_new_name [id=bc87b3ebaa8c4f1b858fb14c03cb7504]) successfully deleted
I20240502 21:26:43.871246 31647 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:26:43.872622 2470 consensus_queue.cc:260] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:43.872848 2365 raft_consensus.cc:482] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:43.873193 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet 0645323dfbad46ceb1e11898968e9d22 (table rebalance_test_table_4 [id=223f9daed2194c19b553fe65892428cf]) successfully deleted
I20240502 21:26:43.872977 2365 raft_consensus.cc:504] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:43.875304 2365 leader_election.cc:288] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:43.876914 2470 ts_tablet_manager.cc:1398] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 21:26:43.877594 2470 tablet_bootstrap.cc:492] T 49acc5301b9849e9b6795179acbc59a8 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:43.877588 1814 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:43.877904 2468 tablet_bootstrap.cc:492] T f02f83ec0a494da5b66241d9b379a03f P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:43.878090 2468 ts_tablet_manager.cc:1367] T f02f83ec0a494da5b66241d9b379a03f P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.098s user 0.007s sys 0.015s
I20240502 21:26:43.879230 2466 tablet_bootstrap.cc:492] T 530f72c556e84bdbbf78fd5d29b6e4db P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:26:43.879441 2466 ts_tablet_manager.cc:1367] T 530f72c556e84bdbbf78fd5d29b6e4db P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.037s user 0.000s sys 0.010s
I20240502 21:26:43.879981 2479 ts_tablet_manager.cc:1888] T e2006432b1c94d4f91cc91b213a28934 P 1ade0354c44c4c37ae480c7a94b11e1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:43.880174 2479 log.cc:1196] T e2006432b1c94d4f91cc91b213a28934 P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal/wals/e2006432b1c94d4f91cc91b213a28934
I20240502 21:26:43.880158 2466 raft_consensus.cc:348] T 530f72c556e84bdbbf78fd5d29b6e4db P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:43.880857 2466 raft_consensus.cc:374] T 530f72c556e84bdbbf78fd5d29b6e4db P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:43.880985 2466 raft_consensus.cc:729] T 530f72c556e84bdbbf78fd5d29b6e4db P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:26:43.881808 2470 tablet_bootstrap.cc:654] T 49acc5301b9849e9b6795179acbc59a8 P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:43.884318 2474 tablet_bootstrap.cc:492] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:43.884943 2466 consensus_queue.cc:260] T 530f72c556e84bdbbf78fd5d29b6e4db P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:43.885502 2466 ts_tablet_manager.cc:1398] T 530f72c556e84bdbbf78fd5d29b6e4db P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 21:26:43.887104 2474 tablet_bootstrap.cc:654] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:43.884413 2479 ts_tablet_manager.cc:1909] T e2006432b1c94d4f91cc91b213a28934 P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting consensus metadata
I20240502 21:26:43.889633 31381 catalog_manager.cc:4830] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): tablet e2006432b1c94d4f91cc91b213a28934 (table rebalancer_dynamic_table_new_name [id=bc87b3ebaa8c4f1b858fb14c03cb7504]) successfully deleted
I20240502 21:26:43.898432 2474 tablet_bootstrap.cc:492] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:43.898670 2474 ts_tablet_manager.cc:1367] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.004s
I20240502 21:26:43.898909 2468 raft_consensus.cc:348] T f02f83ec0a494da5b66241d9b379a03f P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:43.899158 2468 raft_consensus.cc:374] T f02f83ec0a494da5b66241d9b379a03f P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:43.899251 2468 raft_consensus.cc:729] T f02f83ec0a494da5b66241d9b379a03f P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:43.899578 2468 consensus_queue.cc:260] T f02f83ec0a494da5b66241d9b379a03f P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:43.899636 2474 raft_consensus.cc:348] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:43.900022 2474 raft_consensus.cc:374] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:43.900173 2474 raft_consensus.cc:729] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:43.900516 2474 consensus_queue.cc:260] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:43.904601 2474 ts_tablet_manager.cc:1398] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 21:26:43.921331 31906 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 21:26:43.921773 2470 tablet_bootstrap.cc:492] T 49acc5301b9849e9b6795179acbc59a8 P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:43.923360 1735 raft_consensus.cc:482] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:43.923511 1735 raft_consensus.cc:504] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:43.924312 1735 leader_election.cc:288] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:43.925277 2231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49acc5301b9849e9b6795179acbc59a8" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:26:43.925524 2231 raft_consensus.cc:2454] T 49acc5301b9849e9b6795179acbc59a8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:26:43.926350 2470 ts_tablet_manager.cc:1367] T 49acc5301b9849e9b6795179acbc59a8 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.049s user 0.004s sys 0.002s
I20240502 21:26:43.927189 2470 raft_consensus.cc:348] T 49acc5301b9849e9b6795179acbc59a8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:43.931164 2470 raft_consensus.cc:374] T 49acc5301b9849e9b6795179acbc59a8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:43.931268 2470 raft_consensus.cc:729] T 49acc5301b9849e9b6795179acbc59a8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:43.932468 2470 consensus_queue.cc:260] T 49acc5301b9849e9b6795179acbc59a8 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:43.933737 2470 ts_tablet_manager.cc:1398] T 49acc5301b9849e9b6795179acbc59a8 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 21:26:43.933899 2209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:43.937793 1833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f74ea10ab8a2459794f816707b87f494" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:26:43.938083 1833 raft_consensus.cc:2454] T f74ea10ab8a2459794f816707b87f494 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 0.
I20240502 21:26:43.939177 2032 leader_election.cc:302] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 1e04157b98dd4fad98858c290b73c3db; no voters:
I20240502 21:26:43.939620 2365 raft_consensus.cc:2781] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:43.939725 2365 raft_consensus.cc:482] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:43.939848 2365 raft_consensus.cc:3037] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:43.942413 2365 raft_consensus.cc:504] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:43.943048 2365 leader_election.cc:288] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 election: Requested vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:43.943836 1833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f74ea10ab8a2459794f816707b87f494" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:26:43.944046 1833 raft_consensus.cc:3037] T f74ea10ab8a2459794f816707b87f494 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:43.946444 1833 raft_consensus.cc:2454] T f74ea10ab8a2459794f816707b87f494 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 1.
I20240502 21:26:43.946880 2032 leader_election.cc:302] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 1e04157b98dd4fad98858c290b73c3db; no voters:
I20240502 21:26:43.947091 2385 raft_consensus.cc:482] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:43.947199 2365 raft_consensus.cc:2781] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:43.947280 2385 raft_consensus.cc:504] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:43.949867 2385 leader_election.cc:288] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:43.951444 2359 raft_consensus.cc:482] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:43.951624 2359 raft_consensus.cc:504] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:43.952524 2359 leader_election.cc:288] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:26:43.955404 2231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f74ea10ab8a2459794f816707b87f494" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:26:43.956322 2231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f74ea10ab8a2459794f816707b87f494" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
W20240502 21:26:43.957033 2032 leader_election.cc:341] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:26:43.957716 2032 leader_election.cc:341] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:43.960505 2468 ts_tablet_manager.cc:1398] T f02f83ec0a494da5b66241d9b379a03f P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.082s user 0.001s sys 0.001s
I20240502 21:26:43.966132 2468 tablet_bootstrap.cc:492] T f74ea10ab8a2459794f816707b87f494 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:43.973472 31515 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:26:43.978945 2468 tablet_bootstrap.cc:654] T f74ea10ab8a2459794f816707b87f494 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:43.999825 1966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "530f72c556e84bdbbf78fd5d29b6e4db" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:26:44.000074 1966 raft_consensus.cc:2454] T 530f72c556e84bdbbf78fd5d29b6e4db P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 0.
I20240502 21:26:44.009958 1767 leader_election.cc:302] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:26:44.010331 2356 raft_consensus.cc:2781] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:44.010509 2356 raft_consensus.cc:482] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:44.010608 2356 raft_consensus.cc:3037] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:44.013391 2356 raft_consensus.cc:504] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:44.014405 2356 leader_election.cc:288] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Requested vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:26:44.014987 1966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "530f72c556e84bdbbf78fd5d29b6e4db" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf"
I20240502 21:26:44.015182 1966 raft_consensus.cc:3037] T 530f72c556e84bdbbf78fd5d29b6e4db P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:44.018347 31602 leader_election.cc:302] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:44.018669 1735 raft_consensus.cc:2781] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:44.018802 1735 raft_consensus.cc:482] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:44.018899 1735 raft_consensus.cc:3037] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:44.019124 1966 raft_consensus.cc:2454] T 530f72c556e84bdbbf78fd5d29b6e4db P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:26:44.021602 1735 raft_consensus.cc:504] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:44.022173 1735 leader_election.cc:288] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Requested vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:44.022544 1767 leader_election.cc:302] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:26:44.023023 2356 raft_consensus.cc:2781] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:44.023571 2356 raft_consensus.cc:686] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Becoming Leader. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Running, Role: LEADER
I20240502 21:26:44.024096 2356 consensus_queue.cc:237] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:44.024603 2231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02f83ec0a494da5b66241d9b379a03f" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:26:44.024839 2231 raft_consensus.cc:2454] T f02f83ec0a494da5b66241d9b379a03f P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 0.
I20240502 21:26:44.025370 1901 leader_election.cc:302] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dec645b473f40578e726f3e50e219bf, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:44.025753 2385 raft_consensus.cc:2781] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:44.025858 2385 raft_consensus.cc:482] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:44.025946 2385 raft_consensus.cc:3037] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:44.028050 2231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49acc5301b9849e9b6795179acbc59a8" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:26:44.028285 2231 raft_consensus.cc:3037] T 49acc5301b9849e9b6795179acbc59a8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:44.028363 2365 raft_consensus.cc:686] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEADER]: Becoming Leader. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Running, Role: LEADER
I20240502 21:26:44.028798 2385 raft_consensus.cc:504] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:44.028839 2365 consensus_queue.cc:237] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:44.029570 2385 leader_election.cc:288] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 election: Requested vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:44.031181 2506 raft_consensus.cc:482] T 49acc5301b9849e9b6795179acbc59a8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:44.031333 2506 raft_consensus.cc:504] T 49acc5301b9849e9b6795179acbc59a8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:44.033285 2506 leader_election.cc:288] T 49acc5301b9849e9b6795179acbc59a8 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:44.035795 2096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02f83ec0a494da5b66241d9b379a03f" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:44.035991 2096 raft_consensus.cc:2454] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 0.
I20240502 21:26:44.036804 2097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02f83ec0a494da5b66241d9b379a03f" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:26:44.036978 2097 raft_consensus.cc:3037] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:44.036953 2231 raft_consensus.cc:2454] T 49acc5301b9849e9b6795179acbc59a8 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:44.034256 2230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02f83ec0a494da5b66241d9b379a03f" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:26:44.037987 2230 raft_consensus.cc:3037] T f02f83ec0a494da5b66241d9b379a03f P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:44.039801 2097 raft_consensus.cc:2454] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 1.
I20240502 21:26:44.037642 2228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49acc5301b9849e9b6795179acbc59a8" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:26:44.042060 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49acc5301b9849e9b6795179acbc59a8" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:44.042265 2228 raft_consensus.cc:2379] T 49acc5301b9849e9b6795179acbc59a8 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ade0354c44c4c37ae480c7a94b11e1b in current term 1: Already voted for candidate 05f4fac41c4f4959b9b316cb112d0ada in this term.
I20240502 21:26:44.042366 31666 raft_consensus.cc:2379] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ade0354c44c4c37ae480c7a94b11e1b in current term 1: Already voted for candidate 05f4fac41c4f4959b9b316cb112d0ada in this term.
I20240502 21:26:44.043083 2032 leader_election.cc:302] T 49acc5301b9849e9b6795179acbc59a8 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b; no voters: 05f4fac41c4f4959b9b316cb112d0ada, d1406e92aec84ac585304e57bd37a40d
I20240502 21:26:44.043295 2230 raft_consensus.cc:2454] T f02f83ec0a494da5b66241d9b379a03f P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 1.
I20240502 21:26:44.043517 1901 leader_election.cc:302] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:26:44.043807 2506 raft_consensus.cc:3037] T 49acc5301b9849e9b6795179acbc59a8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:44.044112 2385 raft_consensus.cc:2781] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:44.044430 2468 tablet_bootstrap.cc:492] T f74ea10ab8a2459794f816707b87f494 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:44.044610 2468 ts_tablet_manager.cc:1367] T f74ea10ab8a2459794f816707b87f494 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.079s user 0.004s sys 0.000s
I20240502 21:26:44.045387 2468 raft_consensus.cc:348] T f74ea10ab8a2459794f816707b87f494 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:44.045573 2468 raft_consensus.cc:374] T f74ea10ab8a2459794f816707b87f494 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:44.045671 2468 raft_consensus.cc:729] T f74ea10ab8a2459794f816707b87f494 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:44.045879 2468 consensus_queue.cc:260] T f74ea10ab8a2459794f816707b87f494 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:44.046250 31602 leader_election.cc:302] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:44.046514 2097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49acc5301b9849e9b6795179acbc59a8" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:44.046648 2096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49acc5301b9849e9b6795179acbc59a8" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:26:44.047842 1735 raft_consensus.cc:2781] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:44.048079 1735 raft_consensus.cc:686] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Becoming Leader. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Running, Role: LEADER
I20240502 21:26:44.048179 2506 raft_consensus.cc:2726] T 49acc5301b9849e9b6795179acbc59a8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:26:44.048368 1735 consensus_queue.cc:237] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:44.049595 2468 ts_tablet_manager.cc:1398] T f74ea10ab8a2459794f816707b87f494 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 21:26:44.050844 2097 raft_consensus.cc:2454] T 49acc5301b9849e9b6795179acbc59a8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:44.051849 2468 tablet_bootstrap.cc:492] T 530f72c556e84bdbbf78fd5d29b6e4db P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:44.052367 31390 catalog_manager.cc:5484] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130). New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:44.053367 2468 tablet_bootstrap.cc:654] T 530f72c556e84bdbbf78fd5d29b6e4db P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:44.056994 2385 raft_consensus.cc:686] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf [term 1 LEADER]: Becoming Leader. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Running, Role: LEADER
I20240502 21:26:44.057556 2385 consensus_queue.cc:237] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:44.057956 2076 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:44.060201 31390 catalog_manager.cc:5484] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e04157b98dd4fad98858c290b73c3db (127.24.226.133). New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:44.063459 2429 raft_consensus.cc:482] T f74ea10ab8a2459794f816707b87f494 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:44.063620 2429 raft_consensus.cc:504] T f74ea10ab8a2459794f816707b87f494 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:44.064384 2228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "530f72c556e84bdbbf78fd5d29b6e4db" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:26:44.064325 31394 catalog_manager.cc:5484] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135). New cstate: current_term: 1 leader_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:44.065564 2429 leader_election.cc:288] T f74ea10ab8a2459794f816707b87f494 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:44.066856 2230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "530f72c556e84bdbbf78fd5d29b6e4db" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
W20240502 21:26:44.069142 1769 leader_election.cc:341] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:26:44.070066 1769 leader_election.cc:341] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:44.088630 31390 catalog_manager.cc:5484] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dec645b473f40578e726f3e50e219bf (127.24.226.134). New cstate: current_term: 1 leader_uuid: "9dec645b473f40578e726f3e50e219bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:44.109218 2468 tablet_bootstrap.cc:492] T 530f72c556e84bdbbf78fd5d29b6e4db P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:44.109462 2468 ts_tablet_manager.cc:1367] T 530f72c556e84bdbbf78fd5d29b6e4db P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.058s user 0.000s sys 0.004s
I20240502 21:26:44.110234 2468 raft_consensus.cc:348] T 530f72c556e84bdbbf78fd5d29b6e4db P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:44.110535 2468 raft_consensus.cc:374] T 530f72c556e84bdbbf78fd5d29b6e4db P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:44.110698 2468 raft_consensus.cc:729] T 530f72c556e84bdbbf78fd5d29b6e4db P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:44.110985 2468 consensus_queue.cc:260] T 530f72c556e84bdbbf78fd5d29b6e4db P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:44.111713 2468 ts_tablet_manager.cc:1398] T 530f72c556e84bdbbf78fd5d29b6e4db P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:44.112883 1833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f74ea10ab8a2459794f816707b87f494" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:26:44.113219 1833 raft_consensus.cc:2379] T f74ea10ab8a2459794f816707b87f494 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1406e92aec84ac585304e57bd37a40d in current term 1: Already voted for candidate 1ade0354c44c4c37ae480c7a94b11e1b in this term.
I20240502 21:26:44.122339 2097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f74ea10ab8a2459794f816707b87f494" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:44.122967 2163 leader_election.cc:302] T f74ea10ab8a2459794f816707b87f494 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1406e92aec84ac585304e57bd37a40d; no voters: 1ade0354c44c4c37ae480c7a94b11e1b, 1e04157b98dd4fad98858c290b73c3db
I20240502 21:26:44.123381 2429 raft_consensus.cc:3037] T f74ea10ab8a2459794f816707b87f494 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:44.126276 2429 raft_consensus.cc:2726] T f74ea10ab8a2459794f816707b87f494 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:26:44.169435 31390 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240502 21:26:44.169695 31390 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240502 21:26:44.172971 31390 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet b0f8779efe5549f59db5e28324df8e16
I20240502 21:26:44.173527 31390 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet e49ebdbe2bde42bba432662a1c7e7833
I20240502 21:26:44.173940 31390 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 75a35592c4b4495497967bbf256d27af
I20240502 21:26:44.174240 31775 tablet_service.cc:1514] Processing DeleteTablet for tablet e49ebdbe2bde42bba432662a1c7e7833 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:44 UTC) from {username='slave'} at 127.0.0.1:51596
I20240502 21:26:44.174499 2331 tablet_replica.cc:330] T e49ebdbe2bde42bba432662a1c7e7833 P 08d7a0680d244b55b93d17f9ab57b84c: stopping tablet replica
I20240502 21:26:44.174726 2331 raft_consensus.cc:2229] T e49ebdbe2bde42bba432662a1c7e7833 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:44.174901 31515 tablet_service.cc:1514] Processing DeleteTablet for tablet b0f8779efe5549f59db5e28324df8e16 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:44 UTC) from {username='slave'} at 127.0.0.1:45194
I20240502 21:26:44.174957 2331 raft_consensus.cc:2258] T e49ebdbe2bde42bba432662a1c7e7833 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:44.175002 31516 tablet_service.cc:1514] Processing DeleteTablet for tablet e49ebdbe2bde42bba432662a1c7e7833 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:44 UTC) from {username='slave'} at 127.0.0.1:45194
I20240502 21:26:44.175194 2329 tablet_replica.cc:330] T b0f8779efe5549f59db5e28324df8e16 P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:44.175417 2329 raft_consensus.cc:2229] T b0f8779efe5549f59db5e28324df8e16 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:44.175417 31647 tablet_service.cc:1514] Processing DeleteTablet for tablet b0f8779efe5549f59db5e28324df8e16 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:44 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:44.175614 2329 raft_consensus.cc:2258] T b0f8779efe5549f59db5e28324df8e16 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:44.175715 2331 ts_tablet_manager.cc:1875] T e49ebdbe2bde42bba432662a1c7e7833 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:44.175806 31905 tablet_service.cc:1514] Processing DeleteTablet for tablet e49ebdbe2bde42bba432662a1c7e7833 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:44 UTC) from {username='slave'} at 127.0.0.1:47824
I20240502 21:26:44.176198 2328 tablet_replica.cc:330] T e49ebdbe2bde42bba432662a1c7e7833 P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:26:44.176317 2329 ts_tablet_manager.cc:1875] T b0f8779efe5549f59db5e28324df8e16 P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:44.176416 2328 raft_consensus.cc:2229] T e49ebdbe2bde42bba432662a1c7e7833 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:44.176755 2328 raft_consensus.cc:2258] T e49ebdbe2bde42bba432662a1c7e7833 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:44.177423 2328 ts_tablet_manager.cc:1875] T e49ebdbe2bde42bba432662a1c7e7833 P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:44.178964 2330 tablet_replica.cc:330] T b0f8779efe5549f59db5e28324df8e16 P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:44.179134 2330 raft_consensus.cc:2229] T b0f8779efe5549f59db5e28324df8e16 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:44.179353 2330 raft_consensus.cc:2258] T b0f8779efe5549f59db5e28324df8e16 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:44.179824 2330 ts_tablet_manager.cc:1875] T b0f8779efe5549f59db5e28324df8e16 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:44.180567 31906 tablet_service.cc:1514] Processing DeleteTablet for tablet b0f8779efe5549f59db5e28324df8e16 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:44 UTC) from {username='slave'} at 127.0.0.1:47824
I20240502 21:26:44.183164 2331 ts_tablet_manager.cc:1888] T e49ebdbe2bde42bba432662a1c7e7833 P 08d7a0680d244b55b93d17f9ab57b84c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:44.183274 2329 ts_tablet_manager.cc:1888] T b0f8779efe5549f59db5e28324df8e16 P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:44.183429 2329 log.cc:1196] T b0f8779efe5549f59db5e28324df8e16 P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/b0f8779efe5549f59db5e28324df8e16
I20240502 21:26:44.183449 2331 log.cc:1196] T e49ebdbe2bde42bba432662a1c7e7833 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/wals/e49ebdbe2bde42bba432662a1c7e7833
I20240502 21:26:44.183962 2329 ts_tablet_manager.cc:1909] T b0f8779efe5549f59db5e28324df8e16 P a44f41b0b6644fb6aa29476aed8c964f: Deleting consensus metadata
I20240502 21:26:44.184156 2331 ts_tablet_manager.cc:1909] T e49ebdbe2bde42bba432662a1c7e7833 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting consensus metadata
I20240502 21:26:44.184258 31647 tablet_service.cc:1514] Processing DeleteTablet for tablet 75a35592c4b4495497967bbf256d27af with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:44 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:44.184505 31906 tablet_service.cc:1514] Processing DeleteTablet for tablet 75a35592c4b4495497967bbf256d27af with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:44 UTC) from {username='slave'} at 127.0.0.1:47824
I20240502 21:26:44.185566 31381 catalog_manager.cc:4830] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): tablet e49ebdbe2bde42bba432662a1c7e7833 (table rebalance_test_table_6 [id=ab47909c63004e5aad56b045fb34543b]) successfully deleted
I20240502 21:26:44.186097 2328 ts_tablet_manager.cc:1888] T e49ebdbe2bde42bba432662a1c7e7833 P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:44.186138 2330 ts_tablet_manager.cc:1888] T b0f8779efe5549f59db5e28324df8e16 P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:44.186225 2328 log.cc:1196] T e49ebdbe2bde42bba432662a1c7e7833 P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/e49ebdbe2bde42bba432662a1c7e7833
I20240502 21:26:44.186264 2330 log.cc:1196] T b0f8779efe5549f59db5e28324df8e16 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/b0f8779efe5549f59db5e28324df8e16
I20240502 21:26:44.186551 2328 ts_tablet_manager.cc:1909] T e49ebdbe2bde42bba432662a1c7e7833 P 2d50f33557944bb480ea32199d9ee4f1: Deleting consensus metadata
I20240502 21:26:44.186801 2330 ts_tablet_manager.cc:1909] T b0f8779efe5549f59db5e28324df8e16 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:26:44.187325 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet e49ebdbe2bde42bba432662a1c7e7833 (table rebalance_test_table_6 [id=ab47909c63004e5aad56b045fb34543b]) successfully deleted
I20240502 21:26:44.187574 2328 tablet_replica.cc:330] T b0f8779efe5549f59db5e28324df8e16 P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:26:44.187685 2330 tablet_replica.cc:330] T 75a35592c4b4495497967bbf256d27af P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:44.187785 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet b0f8779efe5549f59db5e28324df8e16 (table rebalance_test_table_6 [id=ab47909c63004e5aad56b045fb34543b]) successfully deleted
I20240502 21:26:44.187880 2330 raft_consensus.cc:2229] T 75a35592c4b4495497967bbf256d27af P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:44.188069 2328 raft_consensus.cc:2229] T b0f8779efe5549f59db5e28324df8e16 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:44.188083 31516 tablet_service.cc:1514] Processing DeleteTablet for tablet 75a35592c4b4495497967bbf256d27af with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:44 UTC) from {username='slave'} at 127.0.0.1:45194
I20240502 21:26:44.188207 2330 raft_consensus.cc:2258] T 75a35592c4b4495497967bbf256d27af P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:44.188284 2328 raft_consensus.cc:2258] T b0f8779efe5549f59db5e28324df8e16 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:44.188805 2328 ts_tablet_manager.cc:1875] T b0f8779efe5549f59db5e28324df8e16 P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:44.188813 2330 ts_tablet_manager.cc:1875] T 75a35592c4b4495497967bbf256d27af P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:44.189091 2329 tablet_replica.cc:330] T e49ebdbe2bde42bba432662a1c7e7833 P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:44.189131 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet b0f8779efe5549f59db5e28324df8e16 (table rebalance_test_table_6 [id=ab47909c63004e5aad56b045fb34543b]) successfully deleted
I20240502 21:26:44.189297 2329 raft_consensus.cc:2229] T e49ebdbe2bde42bba432662a1c7e7833 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:44.189556 2329 raft_consensus.cc:2258] T e49ebdbe2bde42bba432662a1c7e7833 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:44.190181 2329 ts_tablet_manager.cc:1875] T e49ebdbe2bde42bba432662a1c7e7833 P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:44.193639 2328 ts_tablet_manager.cc:1888] T b0f8779efe5549f59db5e28324df8e16 P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:44.193825 2328 log.cc:1196] T b0f8779efe5549f59db5e28324df8e16 P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/b0f8779efe5549f59db5e28324df8e16
I20240502 21:26:44.194377 2328 ts_tablet_manager.cc:1909] T b0f8779efe5549f59db5e28324df8e16 P 2d50f33557944bb480ea32199d9ee4f1: Deleting consensus metadata
I20240502 21:26:44.194656 2330 ts_tablet_manager.cc:1888] T 75a35592c4b4495497967bbf256d27af P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:44.194775 2330 log.cc:1196] T 75a35592c4b4495497967bbf256d27af P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/75a35592c4b4495497967bbf256d27af
I20240502 21:26:44.195261 2330 ts_tablet_manager.cc:1909] T 75a35592c4b4495497967bbf256d27af P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:26:44.195452 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet b0f8779efe5549f59db5e28324df8e16 (table rebalance_test_table_6 [id=ab47909c63004e5aad56b045fb34543b]) successfully deleted
I20240502 21:26:44.195714 2328 tablet_replica.cc:330] T 75a35592c4b4495497967bbf256d27af P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:26:44.195886 2328 raft_consensus.cc:2229] T 75a35592c4b4495497967bbf256d27af P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:44.196408 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet 75a35592c4b4495497967bbf256d27af (table rebalance_test_table_6 [id=ab47909c63004e5aad56b045fb34543b]) successfully deleted
I20240502 21:26:44.196710 31390 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:26:44.198784 2328 raft_consensus.cc:2258] T 75a35592c4b4495497967bbf256d27af P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:44.199487 2328 ts_tablet_manager.cc:1875] T 75a35592c4b4495497967bbf256d27af P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:44.202083 2329 ts_tablet_manager.cc:1888] T e49ebdbe2bde42bba432662a1c7e7833 P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:44.202239 2329 log.cc:1196] T e49ebdbe2bde42bba432662a1c7e7833 P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/e49ebdbe2bde42bba432662a1c7e7833
I20240502 21:26:44.202745 2329 ts_tablet_manager.cc:1909] T e49ebdbe2bde42bba432662a1c7e7833 P a44f41b0b6644fb6aa29476aed8c964f: Deleting consensus metadata
I20240502 21:26:44.203963 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet e49ebdbe2bde42bba432662a1c7e7833 (table rebalance_test_table_6 [id=ab47909c63004e5aad56b045fb34543b]) successfully deleted
I20240502 21:26:44.204386 2329 tablet_replica.cc:330] T 75a35592c4b4495497967bbf256d27af P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:44.204654 2329 raft_consensus.cc:2229] T 75a35592c4b4495497967bbf256d27af P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:44.204911 2329 raft_consensus.cc:2258] T 75a35592c4b4495497967bbf256d27af P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:44.205615 2329 ts_tablet_manager.cc:1875] T 75a35592c4b4495497967bbf256d27af P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:44.210810 1966 raft_consensus.cc:1261] T 530f72c556e84bdbbf78fd5d29b6e4db P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:44.210938 2230 raft_consensus.cc:3037] T 530f72c556e84bdbbf78fd5d29b6e4db P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:44.212119 2359 consensus_queue.cc:1035] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:44.212764 2329 ts_tablet_manager.cc:1888] T 75a35592c4b4495497967bbf256d27af P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:44.212921 2329 log.cc:1196] T 75a35592c4b4495497967bbf256d27af P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/75a35592c4b4495497967bbf256d27af
I20240502 21:26:44.213428 2329 ts_tablet_manager.cc:1909] T 75a35592c4b4495497967bbf256d27af P a44f41b0b6644fb6aa29476aed8c964f: Deleting consensus metadata
I20240502 21:26:44.214311 2230 raft_consensus.cc:1261] T 530f72c556e84bdbbf78fd5d29b6e4db P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:44.214574 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet 75a35592c4b4495497967bbf256d27af (table rebalance_test_table_6 [id=ab47909c63004e5aad56b045fb34543b]) successfully deleted
I20240502 21:26:44.215016 2359 consensus_queue.cc:1035] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:44.215973 2328 ts_tablet_manager.cc:1888] T 75a35592c4b4495497967bbf256d27af P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:44.216120 2328 log.cc:1196] T 75a35592c4b4495497967bbf256d27af P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/75a35592c4b4495497967bbf256d27af
I20240502 21:26:44.216655 2328 ts_tablet_manager.cc:1909] T 75a35592c4b4495497967bbf256d27af P 2d50f33557944bb480ea32199d9ee4f1: Deleting consensus metadata
I20240502 21:26:44.218119 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet 75a35592c4b4495497967bbf256d27af (table rebalance_test_table_6 [id=ab47909c63004e5aad56b045fb34543b]) successfully deleted
I20240502 21:26:44.228618 31390 catalog_manager.cc:5576] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753) does not have the latest schema for tablet 530f72c556e84bdbbf78fd5d29b6e4db (table rebalancer_dynamic_table [id=acb3e4c884724fdab17608b6d200b2e3]). Expected version 1 got 0
I20240502 21:26:44.230275 2533 tablet.cc:1722] T 530f72c556e84bdbbf78fd5d29b6e4db P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:44.231072 2533 tablet.cc:1620] T 530f72c556e84bdbbf78fd5d29b6e4db P 9dec645b473f40578e726f3e50e219bf: MemRowSet was empty: no flush needed.
I20240502 21:26:44.233197 31390 catalog_manager.cc:5576] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753) does not have the latest schema for tablet 530f72c556e84bdbbf78fd5d29b6e4db (table rebalancer_dynamic_table [id=acb3e4c884724fdab17608b6d200b2e3]). Expected version 1 got 0
I20240502 21:26:44.237946 2532 tablet.cc:1722] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:44.238615 2532 tablet.cc:1620] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db: MemRowSet was empty: no flush needed.
I20240502 21:26:44.238644 31390 catalog_manager.cc:5576] TS d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907) does not have the latest schema for tablet 530f72c556e84bdbbf78fd5d29b6e4db (table rebalancer_dynamic_table [id=acb3e4c884724fdab17608b6d200b2e3]). Expected version 1 got 0
I20240502 21:26:44.242111 31394 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=acb3e4c884724fdab17608b6d200b2e3] alter complete (version 1)
I20240502 21:26:44.252502 2535 tablet.cc:1722] T 530f72c556e84bdbbf78fd5d29b6e4db P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:44.253962 31394 catalog_manager.cc:5576] TS d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907) does not have the latest schema for tablet 530f72c556e84bdbbf78fd5d29b6e4db (table rebalancer_dynamic_table [id=acb3e4c884724fdab17608b6d200b2e3]). Expected version 1 got 0
I20240502 21:26:44.254045 2535 tablet.cc:1620] T 530f72c556e84bdbbf78fd5d29b6e4db P d1406e92aec84ac585304e57bd37a40d: MemRowSet was empty: no flush needed.
I20240502 21:26:44.353998 31394 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:26:44.363111 2532 tablet.cc:1722] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:44.363631 2532 tablet.cc:1620] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db: MemRowSet was empty: no flush needed.
I20240502 21:26:44.364094 2533 tablet.cc:1722] T 530f72c556e84bdbbf78fd5d29b6e4db P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:44.364421 2535 tablet.cc:1722] T 530f72c556e84bdbbf78fd5d29b6e4db P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:44.364600 2533 tablet.cc:1620] T 530f72c556e84bdbbf78fd5d29b6e4db P 9dec645b473f40578e726f3e50e219bf: MemRowSet was empty: no flush needed.
I20240502 21:26:44.364861 2535 tablet.cc:1620] T 530f72c556e84bdbbf78fd5d29b6e4db P d1406e92aec84ac585304e57bd37a40d: MemRowSet was empty: no flush needed.
I20240502 21:26:44.374696 31394 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=acb3e4c884724fdab17608b6d200b2e3] alter complete (version 2)
I20240502 21:26:44.452899 2365 consensus_queue.cc:1035] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:44.478865 31391 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:26:44.482774 2579 consensus_queue.cc:1035] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:44.484592 31775 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 21:26:44.491747 2532 tablet.cc:1722] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:44.496203 2535 tablet.cc:1722] T 530f72c556e84bdbbf78fd5d29b6e4db P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:44.497073 2533 tablet.cc:1722] T 530f72c556e84bdbbf78fd5d29b6e4db P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:44.501619 31391 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table_new_name [id=acb3e4c884724fdab17608b6d200b2e3] alter complete (version 3)
I20240502 21:26:44.507676 31647 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:26:44.536494 1735 consensus_queue.cc:1035] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:44.539877 31906 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:26:44.554792 2515 consensus_queue.cc:1035] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:44.568471 1813 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:44.575865 2385 consensus_queue.cc:1035] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:44.586629 1946 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:44.598358 31516 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 21:26:44.606719 31390 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:44.606985 31390 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:44.611070 31390 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 530f72c556e84bdbbf78fd5d29b6e4db
I20240502 21:26:44.613055 2209 tablet_service.cc:1514] Processing DeleteTablet for tablet 530f72c556e84bdbbf78fd5d29b6e4db with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:44 UTC) from {username='slave'} at 127.0.0.1:36984
I20240502 21:26:44.613745 1813 tablet_service.cc:1514] Processing DeleteTablet for tablet 530f72c556e84bdbbf78fd5d29b6e4db with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:44 UTC) from {username='slave'} at 127.0.0.1:47510
I20240502 21:26:44.616866 1946 tablet_service.cc:1514] Processing DeleteTablet for tablet 530f72c556e84bdbbf78fd5d29b6e4db with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:44 UTC) from {username='slave'} at 127.0.0.1:33002
I20240502 21:26:44.650804 1735 consensus_queue.cc:1035] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:44.650622 31390 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_extra_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:44.658689 2595 tablet_replica.cc:330] T 530f72c556e84bdbbf78fd5d29b6e4db P 9dec645b473f40578e726f3e50e219bf: stopping tablet replica
I20240502 21:26:44.658914 2595 raft_consensus.cc:2229] T 530f72c556e84bdbbf78fd5d29b6e4db P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:44.659233 2595 raft_consensus.cc:2258] T 530f72c556e84bdbbf78fd5d29b6e4db P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:44.660625 2593 tablet_replica.cc:330] T 530f72c556e84bdbbf78fd5d29b6e4db P d1406e92aec84ac585304e57bd37a40d: stopping tablet replica
I20240502 21:26:44.660826 2593 raft_consensus.cc:2229] T 530f72c556e84bdbbf78fd5d29b6e4db P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:44.661067 2593 raft_consensus.cc:2258] T 530f72c556e84bdbbf78fd5d29b6e4db P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:44.664763 2593 ts_tablet_manager.cc:1875] T 530f72c556e84bdbbf78fd5d29b6e4db P d1406e92aec84ac585304e57bd37a40d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:44.666846 2209 tablet_service.cc:1467] Processing CreateTablet for tablet 579fc086d7524e488f8dc4526b89597b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=5fea5cf0a15d489c9c9a523859a65476]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:44.667253 2210 tablet_service.cc:1467] Processing CreateTablet for tablet 1022c4c6ee8642ca95250933a940fe7a (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=5fea5cf0a15d489c9c9a523859a65476]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:44.667551 1814 tablet_service.cc:1467] Processing CreateTablet for tablet 579fc086d7524e488f8dc4526b89597b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=5fea5cf0a15d489c9c9a523859a65476]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:44.668406 2076 tablet_service.cc:1467] Processing CreateTablet for tablet 579fc086d7524e488f8dc4526b89597b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=5fea5cf0a15d489c9c9a523859a65476]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:44.669076 2076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 579fc086d7524e488f8dc4526b89597b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:44.668025 1814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 579fc086d7524e488f8dc4526b89597b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:44.668494 2075 tablet_service.cc:1467] Processing CreateTablet for tablet 1022c4c6ee8642ca95250933a940fe7a (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=5fea5cf0a15d489c9c9a523859a65476]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:44.672839 2075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1022c4c6ee8642ca95250933a940fe7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:44.676343 1812 tablet_service.cc:1467] Processing CreateTablet for tablet 2db9f537acba4424bca97ffc05c6d0b4 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=5fea5cf0a15d489c9c9a523859a65476]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:44.676908 1812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2db9f537acba4424bca97ffc05c6d0b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:44.667270 2209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 579fc086d7524e488f8dc4526b89597b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:44.684664 2210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1022c4c6ee8642ca95250933a940fe7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:44.684770 31394 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240502 21:26:44.684963 31394 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240502 21:26:44.686275 2594 tablet_replica.cc:330] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db: stopping tablet replica
I20240502 21:26:44.686462 2594 raft_consensus.cc:2229] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:44.686767 2594 raft_consensus.cc:2258] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:44.687798 2594 ts_tablet_manager.cc:1875] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:44.689508 2597 tablet_bootstrap.cc:492] T 579fc086d7524e488f8dc4526b89597b P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:44.697016 31394 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 656e698c34394c38a723230d788b702a
I20240502 21:26:44.697237 31906 tablet_service.cc:1467] Processing CreateTablet for tablet 2db9f537acba4424bca97ffc05c6d0b4 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=5fea5cf0a15d489c9c9a523859a65476]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:44.697618 31394 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 930b5fcce62a4040a2e2b0f0f9462477
I20240502 21:26:44.697736 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2db9f537acba4424bca97ffc05c6d0b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:44.698005 31394 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 98e66cb7daff4977927911bbfad93b62
I20240502 21:26:44.702499 31516 tablet_service.cc:1514] Processing DeleteTablet for tablet 656e698c34394c38a723230d788b702a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:44 UTC) from {username='slave'} at 127.0.0.1:45194
I20240502 21:26:44.702661 31515 tablet_service.cc:1514] Processing DeleteTablet for tablet 930b5fcce62a4040a2e2b0f0f9462477 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:44 UTC) from {username='slave'} at 127.0.0.1:45194
I20240502 21:26:44.702801 2329 tablet_replica.cc:330] T 656e698c34394c38a723230d788b702a P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:44.703003 2329 raft_consensus.cc:2229] T 656e698c34394c38a723230d788b702a P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:44.703254 2329 raft_consensus.cc:2258] T 656e698c34394c38a723230d788b702a P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:44.703430 31775 tablet_service.cc:1514] Processing DeleteTablet for tablet 656e698c34394c38a723230d788b702a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:44 UTC) from {username='slave'} at 127.0.0.1:51596
I20240502 21:26:44.703578 31774 tablet_service.cc:1514] Processing DeleteTablet for tablet 98e66cb7daff4977927911bbfad93b62 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:44 UTC) from {username='slave'} at 127.0.0.1:51596
I20240502 21:26:44.703963 2329 ts_tablet_manager.cc:1875] T 656e698c34394c38a723230d788b702a P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:44.704126 31905 tablet_service.cc:1514] Processing DeleteTablet for tablet 930b5fcce62a4040a2e2b0f0f9462477 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:44 UTC) from {username='slave'} at 127.0.0.1:47824
I20240502 21:26:44.704315 31907 tablet_service.cc:1514] Processing DeleteTablet for tablet 98e66cb7daff4977927911bbfad93b62 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:44 UTC) from {username='slave'} at 127.0.0.1:47824
I20240502 21:26:44.705531 31647 tablet_service.cc:1514] Processing DeleteTablet for tablet 656e698c34394c38a723230d788b702a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:44 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:44.705819 31646 tablet_service.cc:1514] Processing DeleteTablet for tablet 98e66cb7daff4977927911bbfad93b62 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:44 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:44.707422 2077 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:44.708189 2077 tablet_service.cc:1467] Processing CreateTablet for tablet 2db9f537acba4424bca97ffc05c6d0b4 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=5fea5cf0a15d489c9c9a523859a65476]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:44.708642 2077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2db9f537acba4424bca97ffc05c6d0b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:44.708675 2328 tablet_replica.cc:330] T 930b5fcce62a4040a2e2b0f0f9462477 P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:26:44.708884 2328 raft_consensus.cc:2229] T 930b5fcce62a4040a2e2b0f0f9462477 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:44.709139 2328 raft_consensus.cc:2258] T 930b5fcce62a4040a2e2b0f0f9462477 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:44.709862 2328 ts_tablet_manager.cc:1875] T 930b5fcce62a4040a2e2b0f0f9462477 P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:44.712626 31645 tablet_service.cc:1514] Processing DeleteTablet for tablet 930b5fcce62a4040a2e2b0f0f9462477 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:44 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:44.714546 31393 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:26:44.719276 2328 ts_tablet_manager.cc:1888] T 930b5fcce62a4040a2e2b0f0f9462477 P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:44.719483 2328 log.cc:1196] T 930b5fcce62a4040a2e2b0f0f9462477 P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/930b5fcce62a4040a2e2b0f0f9462477
I20240502 21:26:44.720081 2328 ts_tablet_manager.cc:1909] T 930b5fcce62a4040a2e2b0f0f9462477 P 2d50f33557944bb480ea32199d9ee4f1: Deleting consensus metadata
I20240502 21:26:44.727131 31907 tablet_service.cc:1467] Processing CreateTablet for tablet bb7211c63ddd450bb260a32dfb6110b5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=85a71ef99c954778a60e20dc5a6b3d15]), partition=
I20240502 21:26:44.727569 31907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb7211c63ddd450bb260a32dfb6110b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:44.731379 2208 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:44.733513 31515 tablet_service.cc:1467] Processing CreateTablet for tablet bb7211c63ddd450bb260a32dfb6110b5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=85a71ef99c954778a60e20dc5a6b3d15]), partition=
I20240502 21:26:44.734566 31515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb7211c63ddd450bb260a32dfb6110b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:44.736081 2605 tablet_bootstrap.cc:492] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:44.739455 2602 tablet_replica.cc:330] T 656e698c34394c38a723230d788b702a P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:44.739665 2602 raft_consensus.cc:2229] T 656e698c34394c38a723230d788b702a P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:44.739912 2602 raft_consensus.cc:2258] T 656e698c34394c38a723230d788b702a P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:44.740614 2602 ts_tablet_manager.cc:1875] T 656e698c34394c38a723230d788b702a P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:44.741185 2605 tablet_bootstrap.cc:654] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:44.744179 1946 tablet_service.cc:1467] Processing CreateTablet for tablet 1022c4c6ee8642ca95250933a940fe7a (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=5fea5cf0a15d489c9c9a523859a65476]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:44.747318 1946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1022c4c6ee8642ca95250933a940fe7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:44.754776 2602 ts_tablet_manager.cc:1888] T 656e698c34394c38a723230d788b702a P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:44.755154 2602 log.cc:1196] T 656e698c34394c38a723230d788b702a P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/656e698c34394c38a723230d788b702a
I20240502 21:26:44.755777 2602 ts_tablet_manager.cc:1909] T 656e698c34394c38a723230d788b702a P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:26:44.757234 2602 tablet_replica.cc:330] T 98e66cb7daff4977927911bbfad93b62 P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:44.758608 2602 raft_consensus.cc:2229] T 98e66cb7daff4977927911bbfad93b62 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:44.759279 2602 raft_consensus.cc:2258] T 98e66cb7daff4977927911bbfad93b62 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:44.759564 2329 ts_tablet_manager.cc:1888] T 656e698c34394c38a723230d788b702a P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:44.759737 2329 log.cc:1196] T 656e698c34394c38a723230d788b702a P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/656e698c34394c38a723230d788b702a
I20240502 21:26:44.760928 2602 ts_tablet_manager.cc:1875] T 98e66cb7daff4977927911bbfad93b62 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:44.766777 2601 tablet_replica.cc:330] T 656e698c34394c38a723230d788b702a P 08d7a0680d244b55b93d17f9ab57b84c: stopping tablet replica
I20240502 21:26:44.768745 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet 656e698c34394c38a723230d788b702a (table rebalance_test_table_8 [id=f3134ee4e058445b955a09231a5e159f]) successfully deleted
I20240502 21:26:44.769899 2601 raft_consensus.cc:2229] T 656e698c34394c38a723230d788b702a P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:44.770380 2601 raft_consensus.cc:2258] T 656e698c34394c38a723230d788b702a P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:44.772675 2329 ts_tablet_manager.cc:1909] T 656e698c34394c38a723230d788b702a P a44f41b0b6644fb6aa29476aed8c964f: Deleting consensus metadata
I20240502 21:26:44.773852 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet 656e698c34394c38a723230d788b702a (table rebalance_test_table_8 [id=f3134ee4e058445b955a09231a5e159f]) successfully deleted
I20240502 21:26:44.778652 2593 ts_tablet_manager.cc:1888] T 530f72c556e84bdbbf78fd5d29b6e4db P d1406e92aec84ac585304e57bd37a40d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:44.778841 2593 log.cc:1196] T 530f72c556e84bdbbf78fd5d29b6e4db P d1406e92aec84ac585304e57bd37a40d: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal/wals/530f72c556e84bdbbf78fd5d29b6e4db
I20240502 21:26:44.779364 2593 ts_tablet_manager.cc:1909] T 530f72c556e84bdbbf78fd5d29b6e4db P d1406e92aec84ac585304e57bd37a40d: Deleting consensus metadata
I20240502 21:26:44.780315 31381 catalog_manager.cc:4830] TS d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): tablet 530f72c556e84bdbbf78fd5d29b6e4db (table rebalancer_dynamic_table_new_name [id=acb3e4c884724fdab17608b6d200b2e3]) successfully deleted
I20240502 21:26:44.781324 2602 ts_tablet_manager.cc:1888] T 98e66cb7daff4977927911bbfad93b62 P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:44.781662 2602 log.cc:1196] T 98e66cb7daff4977927911bbfad93b62 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/98e66cb7daff4977927911bbfad93b62
I20240502 21:26:44.782223 2602 ts_tablet_manager.cc:1909] T 98e66cb7daff4977927911bbfad93b62 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:26:44.782305 2595 ts_tablet_manager.cc:1875] T 530f72c556e84bdbbf78fd5d29b6e4db P 9dec645b473f40578e726f3e50e219bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:44.783643 2602 tablet_replica.cc:330] T 930b5fcce62a4040a2e2b0f0f9462477 P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:44.784808 2602 raft_consensus.cc:2229] T 930b5fcce62a4040a2e2b0f0f9462477 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:44.785194 2328 tablet_replica.cc:330] T 98e66cb7daff4977927911bbfad93b62 P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:26:44.785720 2602 raft_consensus.cc:2258] T 930b5fcce62a4040a2e2b0f0f9462477 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:44.785708 2328 raft_consensus.cc:2229] T 98e66cb7daff4977927911bbfad93b62 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:44.786038 2328 raft_consensus.cc:2258] T 98e66cb7daff4977927911bbfad93b62 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:44.786402 2602 ts_tablet_manager.cc:1875] T 930b5fcce62a4040a2e2b0f0f9462477 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:44.786720 2329 tablet_replica.cc:330] T 930b5fcce62a4040a2e2b0f0f9462477 P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:44.786708 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet 930b5fcce62a4040a2e2b0f0f9462477 (table rebalance_test_table_8 [id=f3134ee4e058445b955a09231a5e159f]) successfully deleted
I20240502 21:26:44.787174 1813 tablet_service.cc:1467] Processing CreateTablet for tablet bb7211c63ddd450bb260a32dfb6110b5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=85a71ef99c954778a60e20dc5a6b3d15]), partition=
I20240502 21:26:44.787549 1813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb7211c63ddd450bb260a32dfb6110b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:44.787943 2328 ts_tablet_manager.cc:1875] T 98e66cb7daff4977927911bbfad93b62 P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:44.788374 2329 raft_consensus.cc:2229] T 930b5fcce62a4040a2e2b0f0f9462477 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:44.792831 2329 raft_consensus.cc:2258] T 930b5fcce62a4040a2e2b0f0f9462477 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:44.795943 2329 ts_tablet_manager.cc:1875] T 930b5fcce62a4040a2e2b0f0f9462477 P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:44.796296 2605 tablet_bootstrap.cc:492] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:44.796510 2605 ts_tablet_manager.cc:1367] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.061s user 0.011s sys 0.010s
I20240502 21:26:44.796559 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet 98e66cb7daff4977927911bbfad93b62 (table rebalance_test_table_8 [id=f3134ee4e058445b955a09231a5e159f]) successfully deleted
I20240502 21:26:44.797631 2611 tablet_bootstrap.cc:492] T 1022c4c6ee8642ca95250933a940fe7a P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:44.797492 2605 raft_consensus.cc:348] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:44.797745 2605 raft_consensus.cc:374] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:44.797905 2605 raft_consensus.cc:729] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:44.798368 2604 tablet_bootstrap.cc:492] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:44.798611 2605 consensus_queue.cc:260] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:44.799401 2605 ts_tablet_manager.cc:1398] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:44.799722 2605 tablet_bootstrap.cc:492] T bb7211c63ddd450bb260a32dfb6110b5 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:44.801280 2605 tablet_bootstrap.cc:654] T bb7211c63ddd450bb260a32dfb6110b5 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:44.801985 2604 tablet_bootstrap.cc:654] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:44.803550 2602 ts_tablet_manager.cc:1888] T 930b5fcce62a4040a2e2b0f0f9462477 P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:44.803753 2602 log.cc:1196] T 930b5fcce62a4040a2e2b0f0f9462477 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/930b5fcce62a4040a2e2b0f0f9462477
I20240502 21:26:44.804279 2602 ts_tablet_manager.cc:1909] T 930b5fcce62a4040a2e2b0f0f9462477 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:26:44.806046 2606 tablet_bootstrap.cc:492] T 1022c4c6ee8642ca95250933a940fe7a P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:44.807686 2595 ts_tablet_manager.cc:1888] T 530f72c556e84bdbbf78fd5d29b6e4db P 9dec645b473f40578e726f3e50e219bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:44.808094 2606 tablet_bootstrap.cc:654] T 1022c4c6ee8642ca95250933a940fe7a P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:44.808105 2605 tablet_bootstrap.cc:492] T bb7211c63ddd450bb260a32dfb6110b5 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:44.808259 2605 ts_tablet_manager.cc:1367] T bb7211c63ddd450bb260a32dfb6110b5 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240502 21:26:44.808627 2611 tablet_bootstrap.cc:654] T 1022c4c6ee8642ca95250933a940fe7a P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:44.809295 2605 raft_consensus.cc:348] T bb7211c63ddd450bb260a32dfb6110b5 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:44.809567 2605 raft_consensus.cc:374] T bb7211c63ddd450bb260a32dfb6110b5 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:44.809708 2605 raft_consensus.cc:729] T bb7211c63ddd450bb260a32dfb6110b5 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:44.810043 2605 consensus_queue.cc:260] T bb7211c63ddd450bb260a32dfb6110b5 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:44.810320 2595 log.cc:1196] T 530f72c556e84bdbbf78fd5d29b6e4db P 9dec645b473f40578e726f3e50e219bf: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal/wals/530f72c556e84bdbbf78fd5d29b6e4db
I20240502 21:26:44.810796 2595 ts_tablet_manager.cc:1909] T 530f72c556e84bdbbf78fd5d29b6e4db P 9dec645b473f40578e726f3e50e219bf: Deleting consensus metadata
I20240502 21:26:44.810827 2605 ts_tablet_manager.cc:1398] T bb7211c63ddd450bb260a32dfb6110b5 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:44.811940 31379 catalog_manager.cc:4830] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): tablet 530f72c556e84bdbbf78fd5d29b6e4db (table rebalancer_dynamic_table_new_name [id=acb3e4c884724fdab17608b6d200b2e3]) successfully deleted
I20240502 21:26:44.816020 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet 930b5fcce62a4040a2e2b0f0f9462477 (table rebalance_test_table_8 [id=f3134ee4e058445b955a09231a5e159f]) successfully deleted
I20240502 21:26:44.797420 2610 tablet_bootstrap.cc:492] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:44.827785 2610 tablet_bootstrap.cc:654] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:44.831034 2597 tablet_bootstrap.cc:654] T 579fc086d7524e488f8dc4526b89597b P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:44.775974 2601 ts_tablet_manager.cc:1875] T 656e698c34394c38a723230d788b702a P 08d7a0680d244b55b93d17f9ab57b84c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:44.835673 2328 ts_tablet_manager.cc:1888] T 98e66cb7daff4977927911bbfad93b62 P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:44.835884 2328 log.cc:1196] T 98e66cb7daff4977927911bbfad93b62 P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/98e66cb7daff4977927911bbfad93b62
I20240502 21:26:44.836701 2328 ts_tablet_manager.cc:1909] T 98e66cb7daff4977927911bbfad93b62 P 2d50f33557944bb480ea32199d9ee4f1: Deleting consensus metadata
I20240502 21:26:44.840274 2594 ts_tablet_manager.cc:1888] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:44.841284 2594 log.cc:1196] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal/wals/530f72c556e84bdbbf78fd5d29b6e4db
I20240502 21:26:44.841900 2604 tablet_bootstrap.cc:492] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:44.842087 2604 ts_tablet_manager.cc:1367] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.000s
I20240502 21:26:44.842865 2604 raft_consensus.cc:348] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:44.843120 2604 raft_consensus.cc:374] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:44.843281 2604 raft_consensus.cc:729] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:44.843627 2604 consensus_queue.cc:260] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:44.847340 2601 ts_tablet_manager.cc:1888] T 656e698c34394c38a723230d788b702a P 08d7a0680d244b55b93d17f9ab57b84c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:44.847481 2601 log.cc:1196] T 656e698c34394c38a723230d788b702a P 08d7a0680d244b55b93d17f9ab57b84c: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/wals/656e698c34394c38a723230d788b702a
I20240502 21:26:44.847967 2601 ts_tablet_manager.cc:1909] T 656e698c34394c38a723230d788b702a P 08d7a0680d244b55b93d17f9ab57b84c: Deleting consensus metadata
I20240502 21:26:44.849040 2594 ts_tablet_manager.cc:1909] T 530f72c556e84bdbbf78fd5d29b6e4db P 1e04157b98dd4fad98858c290b73c3db: Deleting consensus metadata
I20240502 21:26:44.853406 2606 tablet_bootstrap.cc:492] T 1022c4c6ee8642ca95250933a940fe7a P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:44.853579 2606 ts_tablet_manager.cc:1367] T 1022c4c6ee8642ca95250933a940fe7a P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.048s user 0.001s sys 0.002s
I20240502 21:26:44.854350 2606 raft_consensus.cc:348] T 1022c4c6ee8642ca95250933a940fe7a P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:44.854528 2606 raft_consensus.cc:374] T 1022c4c6ee8642ca95250933a940fe7a P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:44.854648 2606 raft_consensus.cc:729] T 1022c4c6ee8642ca95250933a940fe7a P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:44.854862 2606 consensus_queue.cc:260] T 1022c4c6ee8642ca95250933a940fe7a P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:44.855463 2606 ts_tablet_manager.cc:1398] T 1022c4c6ee8642ca95250933a940fe7a P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:44.855762 2606 tablet_bootstrap.cc:492] T 579fc086d7524e488f8dc4526b89597b P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:44.858430 2610 tablet_bootstrap.cc:492] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:44.858470 31381 catalog_manager.cc:4830] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): tablet 530f72c556e84bdbbf78fd5d29b6e4db (table rebalancer_dynamic_table_new_name [id=acb3e4c884724fdab17608b6d200b2e3]) successfully deleted
I20240502 21:26:44.858662 2610 ts_tablet_manager.cc:1367] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.061s user 0.004s sys 0.000s
I20240502 21:26:44.859467 2610 raft_consensus.cc:348] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:44.859764 2604 ts_tablet_manager.cc:1398] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240502 21:26:44.860970 2610 raft_consensus.cc:374] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:44.861116 2610 raft_consensus.cc:729] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:44.861378 2610 consensus_queue.cc:260] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:44.861845 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet 98e66cb7daff4977927911bbfad93b62 (table rebalance_test_table_8 [id=f3134ee4e058445b955a09231a5e159f]) successfully deleted
I20240502 21:26:44.861907 2604 tablet_bootstrap.cc:492] T 579fc086d7524e488f8dc4526b89597b P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:44.862234 2610 ts_tablet_manager.cc:1398] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:44.860867 2329 ts_tablet_manager.cc:1888] T 930b5fcce62a4040a2e2b0f0f9462477 P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:44.862591 2329 log.cc:1196] T 930b5fcce62a4040a2e2b0f0f9462477 P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/930b5fcce62a4040a2e2b0f0f9462477
I20240502 21:26:44.863057 2329 ts_tablet_manager.cc:1909] T 930b5fcce62a4040a2e2b0f0f9462477 P a44f41b0b6644fb6aa29476aed8c964f: Deleting consensus metadata
I20240502 21:26:44.863687 2604 tablet_bootstrap.cc:654] T 579fc086d7524e488f8dc4526b89597b P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:44.864949 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet 930b5fcce62a4040a2e2b0f0f9462477 (table rebalance_test_table_8 [id=f3134ee4e058445b955a09231a5e159f]) successfully deleted
I20240502 21:26:44.866606 2611 tablet_bootstrap.cc:492] T 1022c4c6ee8642ca95250933a940fe7a P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:26:44.866822 2611 ts_tablet_manager.cc:1367] T 1022c4c6ee8642ca95250933a940fe7a P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.069s user 0.011s sys 0.032s
I20240502 21:26:44.867761 2611 raft_consensus.cc:348] T 1022c4c6ee8642ca95250933a940fe7a P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:44.868070 2611 raft_consensus.cc:374] T 1022c4c6ee8642ca95250933a940fe7a P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:44.868273 2611 raft_consensus.cc:729] T 1022c4c6ee8642ca95250933a940fe7a P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:26:44.868624 2611 consensus_queue.cc:260] T 1022c4c6ee8642ca95250933a940fe7a P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:44.871412 2611 ts_tablet_manager.cc:1398] T 1022c4c6ee8642ca95250933a940fe7a P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:26:44.873402 2597 tablet_bootstrap.cc:492] T 579fc086d7524e488f8dc4526b89597b P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:44.873615 2597 ts_tablet_manager.cc:1367] T 579fc086d7524e488f8dc4526b89597b P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.184s user 0.004s sys 0.000s
I20240502 21:26:44.874392 2597 raft_consensus.cc:348] T 579fc086d7524e488f8dc4526b89597b P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:44.874617 2597 raft_consensus.cc:374] T 579fc086d7524e488f8dc4526b89597b P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:44.874748 2597 raft_consensus.cc:729] T 579fc086d7524e488f8dc4526b89597b P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:44.874964 2597 consensus_queue.cc:260] T 579fc086d7524e488f8dc4526b89597b P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:44.875936 2597 ts_tablet_manager.cc:1398] T 579fc086d7524e488f8dc4526b89597b P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:44.876259 2597 tablet_bootstrap.cc:492] T 2db9f537acba4424bca97ffc05c6d0b4 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:44.877174 2606 tablet_bootstrap.cc:654] T 579fc086d7524e488f8dc4526b89597b P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:44.884886 2597 tablet_bootstrap.cc:654] T 2db9f537acba4424bca97ffc05c6d0b4 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:44.893872 2597 tablet_bootstrap.cc:492] T 2db9f537acba4424bca97ffc05c6d0b4 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:44.894078 2597 ts_tablet_manager.cc:1367] T 2db9f537acba4424bca97ffc05c6d0b4 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240502 21:26:44.894793 2597 raft_consensus.cc:348] T 2db9f537acba4424bca97ffc05c6d0b4 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:44.895599 2601 tablet_replica.cc:330] T 98e66cb7daff4977927911bbfad93b62 P 08d7a0680d244b55b93d17f9ab57b84c: stopping tablet replica
I20240502 21:26:44.895866 2601 raft_consensus.cc:2229] T 98e66cb7daff4977927911bbfad93b62 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:44.896023 2597 raft_consensus.cc:374] T 2db9f537acba4424bca97ffc05c6d0b4 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:44.896135 2597 raft_consensus.cc:729] T 2db9f537acba4424bca97ffc05c6d0b4 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:44.896488 31381 catalog_manager.cc:4830] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): tablet 656e698c34394c38a723230d788b702a (table rebalance_test_table_8 [id=f3134ee4e058445b955a09231a5e159f]) successfully deleted
I20240502 21:26:44.896754 2597 consensus_queue.cc:260] T 2db9f537acba4424bca97ffc05c6d0b4 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:44.896987 2601 raft_consensus.cc:2258] T 98e66cb7daff4977927911bbfad93b62 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:44.897401 2597 ts_tablet_manager.cc:1398] T 2db9f537acba4424bca97ffc05c6d0b4 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:26:44.898275 2601 ts_tablet_manager.cc:1875] T 98e66cb7daff4977927911bbfad93b62 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:44.898782 2597 tablet_bootstrap.cc:492] T bb7211c63ddd450bb260a32dfb6110b5 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:44.902330 2597 tablet_bootstrap.cc:654] T bb7211c63ddd450bb260a32dfb6110b5 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:44.903532 2604 tablet_bootstrap.cc:492] T 579fc086d7524e488f8dc4526b89597b P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:44.903753 2604 ts_tablet_manager.cc:1367] T 579fc086d7524e488f8dc4526b89597b P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.042s user 0.018s sys 0.019s
I20240502 21:26:44.904546 2604 raft_consensus.cc:348] T 579fc086d7524e488f8dc4526b89597b P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:44.904831 2604 raft_consensus.cc:374] T 579fc086d7524e488f8dc4526b89597b P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:44.904933 2604 raft_consensus.cc:729] T 579fc086d7524e488f8dc4526b89597b P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:44.905236 2604 consensus_queue.cc:260] T 579fc086d7524e488f8dc4526b89597b P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:44.905889 2604 ts_tablet_manager.cc:1398] T 579fc086d7524e488f8dc4526b89597b P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:44.909457 2606 tablet_bootstrap.cc:492] T 579fc086d7524e488f8dc4526b89597b P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:44.909690 2606 ts_tablet_manager.cc:1367] T 579fc086d7524e488f8dc4526b89597b P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.054s user 0.005s sys 0.000s
I20240502 21:26:44.912672 2606 raft_consensus.cc:348] T 579fc086d7524e488f8dc4526b89597b P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:44.912936 2606 raft_consensus.cc:374] T 579fc086d7524e488f8dc4526b89597b P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:44.913048 2606 raft_consensus.cc:729] T 579fc086d7524e488f8dc4526b89597b P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:44.913348 2606 consensus_queue.cc:260] T 579fc086d7524e488f8dc4526b89597b P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:44.914433 2606 ts_tablet_manager.cc:1398] T 579fc086d7524e488f8dc4526b89597b P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:26:44.914732 2606 tablet_bootstrap.cc:492] T 2db9f537acba4424bca97ffc05c6d0b4 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:44.916101 2606 tablet_bootstrap.cc:654] T 2db9f537acba4424bca97ffc05c6d0b4 P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:44.917632 2601 ts_tablet_manager.cc:1888] T 98e66cb7daff4977927911bbfad93b62 P 08d7a0680d244b55b93d17f9ab57b84c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:44.917960 2601 log.cc:1196] T 98e66cb7daff4977927911bbfad93b62 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/wals/98e66cb7daff4977927911bbfad93b62
I20240502 21:26:44.918690 2601 ts_tablet_manager.cc:1909] T 98e66cb7daff4977927911bbfad93b62 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting consensus metadata
I20240502 21:26:44.922030 31381 catalog_manager.cc:4830] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): tablet 98e66cb7daff4977927911bbfad93b62 (table rebalance_test_table_8 [id=f3134ee4e058445b955a09231a5e159f]) successfully deleted
I20240502 21:26:44.925163 2597 tablet_bootstrap.cc:492] T bb7211c63ddd450bb260a32dfb6110b5 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:44.925345 2597 ts_tablet_manager.cc:1367] T bb7211c63ddd450bb260a32dfb6110b5 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.000s
I20240502 21:26:44.926064 2597 raft_consensus.cc:348] T bb7211c63ddd450bb260a32dfb6110b5 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:44.926330 2597 raft_consensus.cc:374] T bb7211c63ddd450bb260a32dfb6110b5 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:44.926461 2597 raft_consensus.cc:729] T bb7211c63ddd450bb260a32dfb6110b5 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:44.926729 2597 consensus_queue.cc:260] T bb7211c63ddd450bb260a32dfb6110b5 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:44.929373 2597 ts_tablet_manager.cc:1398] T bb7211c63ddd450bb260a32dfb6110b5 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 21:26:44.929877 2582 raft_consensus.cc:482] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:44.930032 2582 raft_consensus.cc:504] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:44.930924 2579 raft_consensus.cc:482] T 579fc086d7524e488f8dc4526b89597b P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:44.930970 2582 leader_election.cc:288] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:44.931073 2579 raft_consensus.cc:504] T 579fc086d7524e488f8dc4526b89597b P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:44.931692 2579 leader_election.cc:288] T 579fc086d7524e488f8dc4526b89597b P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:26:44.933230 2228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "579fc086d7524e488f8dc4526b89597b" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:26:44.933470 2228 raft_consensus.cc:2454] T 579fc086d7524e488f8dc4526b89597b P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 0.
I20240502 21:26:44.933877 2032 leader_election.cc:302] T 579fc086d7524e488f8dc4526b89597b P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:44.933884 1833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "579fc086d7524e488f8dc4526b89597b" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:26:44.934159 1833 raft_consensus.cc:2454] T 579fc086d7524e488f8dc4526b89597b P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 0.
I20240502 21:26:44.934265 2579 raft_consensus.cc:2781] T 579fc086d7524e488f8dc4526b89597b P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:44.934347 2579 raft_consensus.cc:482] T 579fc086d7524e488f8dc4526b89597b P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:44.934441 2579 raft_consensus.cc:3037] T 579fc086d7524e488f8dc4526b89597b P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:44.937100 2579 raft_consensus.cc:504] T 579fc086d7524e488f8dc4526b89597b P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:44.940419 2228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "579fc086d7524e488f8dc4526b89597b" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:26:44.940644 2228 raft_consensus.cc:3037] T 579fc086d7524e488f8dc4526b89597b P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:44.941329 2606 tablet_bootstrap.cc:492] T 2db9f537acba4424bca97ffc05c6d0b4 P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:44.941462 2606 ts_tablet_manager.cc:1367] T 2db9f537acba4424bca97ffc05c6d0b4 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.000s
I20240502 21:26:44.942203 2606 raft_consensus.cc:348] T 2db9f537acba4424bca97ffc05c6d0b4 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:44.942409 2606 raft_consensus.cc:374] T 2db9f537acba4424bca97ffc05c6d0b4 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:44.942503 2606 raft_consensus.cc:729] T 2db9f537acba4424bca97ffc05c6d0b4 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:44.942744 2606 consensus_queue.cc:260] T 2db9f537acba4424bca97ffc05c6d0b4 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:44.943226 2228 raft_consensus.cc:2454] T 579fc086d7524e488f8dc4526b89597b P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 1.
I20240502 21:26:44.943662 2032 leader_election.cc:302] T 579fc086d7524e488f8dc4526b89597b P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:44.943953 2365 raft_consensus.cc:2781] T 579fc086d7524e488f8dc4526b89597b P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:44.944056 2606 ts_tablet_manager.cc:1398] T 2db9f537acba4424bca97ffc05c6d0b4 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:44.944093 2365 raft_consensus.cc:686] T 579fc086d7524e488f8dc4526b89597b P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEADER]: Becoming Leader. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Running, Role: LEADER
I20240502 21:26:44.944346 2365 consensus_queue.cc:237] T 579fc086d7524e488f8dc4526b89597b P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:44.947865 2579 leader_election.cc:288] T 579fc086d7524e488f8dc4526b89597b P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 election: Requested vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:26:44.948415 1833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "579fc086d7524e488f8dc4526b89597b" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:26:44.948621 1833 raft_consensus.cc:3037] T 579fc086d7524e488f8dc4526b89597b P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:26:44.951205 31973 tablet.cc:2369] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:26:44.951310 1833 raft_consensus.cc:2454] T 579fc086d7524e488f8dc4526b89597b P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 1.
I20240502 21:26:44.954794 31390 catalog_manager.cc:5484] T 579fc086d7524e488f8dc4526b89597b P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135). New cstate: current_term: 1 leader_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:44.977385 1833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db9f537acba4424bca97ffc05c6d0b4" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:26:44.978025 1833 raft_consensus.cc:2454] T 2db9f537acba4424bca97ffc05c6d0b4 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:44.979132 31862 leader_election.cc:302] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:44.979771 1884 raft_consensus.cc:2781] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:44.979772 2097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db9f537acba4424bca97ffc05c6d0b4" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:44.979863 1884 raft_consensus.cc:482] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:44.979955 1884 raft_consensus.cc:3037] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:44.979965 2097 raft_consensus.cc:2454] T 2db9f537acba4424bca97ffc05c6d0b4 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:44.982342 1884 raft_consensus.cc:504] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:44.982918 1884 leader_election.cc:288] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Requested vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:44.983217 1833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db9f537acba4424bca97ffc05c6d0b4" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:26:44.983485 1833 raft_consensus.cc:3037] T 2db9f537acba4424bca97ffc05c6d0b4 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:44.986194 1833 raft_consensus.cc:2454] T 2db9f537acba4424bca97ffc05c6d0b4 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:26:44.986979 2097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db9f537acba4424bca97ffc05c6d0b4" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:26:44.987159 2097 raft_consensus.cc:3037] T 2db9f537acba4424bca97ffc05c6d0b4 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:44.989758 2097 raft_consensus.cc:2454] T 2db9f537acba4424bca97ffc05c6d0b4 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:26:44.992358 31862 leader_election.cc:302] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:44.992668 1884 raft_consensus.cc:2781] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:44.992831 1884 raft_consensus.cc:686] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Running, Role: LEADER
I20240502 21:26:44.993089 1884 consensus_queue.cc:237] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:44.994200 2429 raft_consensus.cc:482] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:44.994426 2429 raft_consensus.cc:504] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:44.995139 2429 leader_election.cc:288] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:44.995820 2097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1022c4c6ee8642ca95250933a940fe7a" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:44.996045 2097 raft_consensus.cc:2454] T 1022c4c6ee8642ca95250933a940fe7a P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:26:44.997592 2163 leader_election.cc:302] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:44.997936 2429 raft_consensus.cc:2781] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:44.998024 2429 raft_consensus.cc:482] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:44.998126 2429 raft_consensus.cc:3037] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:45.001899 2429 raft_consensus.cc:504] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:45.002578 2429 leader_election.cc:288] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Requested vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:45.003197 2097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1022c4c6ee8642ca95250933a940fe7a" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:26:45.003381 2097 raft_consensus.cc:3037] T 1022c4c6ee8642ca95250933a940fe7a P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:45.004402 31390 catalog_manager.cc:5484] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132). New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:45.005895 2097 raft_consensus.cc:2454] T 1022c4c6ee8642ca95250933a940fe7a P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 1.
I20240502 21:26:45.006479 2163 leader_election.cc:302] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:45.006791 2429 raft_consensus.cc:2781] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:45.013716 2429 raft_consensus.cc:686] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Becoming Leader. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Running, Role: LEADER
I20240502 21:26:45.014472 2429 consensus_queue.cc:237] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:45.015686 1557 raft_consensus.cc:482] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:45.015893 1557 raft_consensus.cc:504] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:45.016825 1557 leader_election.cc:288] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:26:45.025061 31924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb7211c63ddd450bb260a32dfb6110b5" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:45.025293 31924 raft_consensus.cc:2454] T bb7211c63ddd450bb260a32dfb6110b5 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:45.026191 31472 leader_election.cc:302] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:45.026746 2303 raft_consensus.cc:2781] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:45.026953 2303 raft_consensus.cc:482] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:45.027062 2303 raft_consensus.cc:3037] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:45.030984 2303 raft_consensus.cc:504] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:45.032254 31924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb7211c63ddd450bb260a32dfb6110b5" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:45.032532 31924 raft_consensus.cc:3037] T bb7211c63ddd450bb260a32dfb6110b5 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:45.034665 2303 leader_election.cc:288] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Requested vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:26:45.040702 1834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb7211c63ddd450bb260a32dfb6110b5" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:26:45.041004 1834 raft_consensus.cc:3037] T bb7211c63ddd450bb260a32dfb6110b5 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:45.041625 1833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb7211c63ddd450bb260a32dfb6110b5" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:26:45.042243 1966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1022c4c6ee8642ca95250933a940fe7a" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:26:45.042344 1965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1022c4c6ee8642ca95250933a940fe7a" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf"
I20240502 21:26:45.042510 1966 raft_consensus.cc:2454] T 1022c4c6ee8642ca95250933a940fe7a P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:26:45.042984 31390 catalog_manager.cc:5484] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d reported cstate change: term changed from 0 to 1, leader changed from <none> to d1406e92aec84ac585304e57bd37a40d (127.24.226.136). New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:45.043560 1834 raft_consensus.cc:2454] T bb7211c63ddd450bb260a32dfb6110b5 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:45.044394 31924 raft_consensus.cc:2454] T bb7211c63ddd450bb260a32dfb6110b5 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:45.045017 31472 leader_election.cc:302] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:45.045935 2631 raft_consensus.cc:2781] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:45.046248 2631 raft_consensus.cc:686] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:26:45.046649 2631 consensus_queue.cc:237] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:45.050611 31390 catalog_manager.cc:5484] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 0 to 1, leader changed from <none> to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:45.075850 31924 consensus_queue.cc:237] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:26:45.078239 31666 raft_consensus.cc:1261] T bb64e0d78397429d959535f570658594 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:45.078725 31797 raft_consensus.cc:1261] T bb64e0d78397429d959535f570658594 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:45.079183 2582 consensus_queue.cc:1035] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:45.079524 1884 consensus_queue.cc:1035] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:26:45.084952 31860 consensus_peers.cc:485] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1 -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb64e0d78397429d959535f570658594. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:45.084650 1884 raft_consensus.cc:2932] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:45.086683 31797 raft_consensus.cc:2932] T bb64e0d78397429d959535f570658594 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:45.089352 31666 raft_consensus.cc:2932] T bb64e0d78397429d959535f570658594 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:45.092775 31393 catalog_manager.cc:5484] T bb64e0d78397429d959535f570658594 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:45.129078 31666 consensus_queue.cc:237] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } }
I20240502 21:26:45.132405 31797 raft_consensus.cc:1261] T 182b3f15c06c4c07ade0a8854d3ecafd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:45.132532 31534 raft_consensus.cc:1261] T 182b3f15c06c4c07ade0a8854d3ecafd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:45.132961 2603 consensus_queue.cc:1035] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:45.133425 2599 consensus_queue.cc:1035] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:45.137487 2599 raft_consensus.cc:2932] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } } }
I20240502 21:26:45.141116 31534 raft_consensus.cc:2932] T 182b3f15c06c4c07ade0a8854d3ecafd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } } }
I20240502 21:26:45.143878 31390 catalog_manager.cc:5484] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: config changed from index -1 to 4, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 21:26:45.149134 31602 consensus_peers.cc:485] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): Couldn't send request to peer 1e04157b98dd4fad98858c290b73c3db. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 182b3f15c06c4c07ade0a8854d3ecafd. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:45.159135 31797 raft_consensus.cc:2932] T 182b3f15c06c4c07ade0a8854d3ecafd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } } }
I20240502 21:26:45.176183 1965 consensus_queue.cc:237] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } }
I20240502 21:26:45.179200 2097 raft_consensus.cc:1261] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:45.179956 2228 raft_consensus.cc:1261] T f02f83ec0a494da5b66241d9b379a03f P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:45.180532 2515 consensus_queue.cc:1035] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:45.180855 2644 consensus_queue.cc:1035] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:45.183686 2644 raft_consensus.cc:2932] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } } }
I20240502 21:26:45.184190 2097 raft_consensus.cc:2932] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } } }
I20240502 21:26:45.184195 2228 raft_consensus.cc:2932] T f02f83ec0a494da5b66241d9b379a03f P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } } }
I20240502 21:26:45.190387 31393 catalog_manager.cc:5484] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: config changed from index -1 to 2, NON_VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) added. New cstate: current_term: 1 leader_uuid: "9dec645b473f40578e726f3e50e219bf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } } }
W20240502 21:26:45.202183 1902 consensus_peers.cc:485] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf -> Peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Couldn't send request to peer a44f41b0b6644fb6aa29476aed8c964f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f02f83ec0a494da5b66241d9b379a03f. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:45.203126 31393 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240502 21:26:45.203342 31393 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240502 21:26:45.206888 31393 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 900be14c60dc4be0adc22c2bcee4dd81
I20240502 21:26:45.207299 31393 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 9a0d1c43a0794c2884dfef498f046f6d
I20240502 21:26:45.207887 31647 tablet_service.cc:1514] Processing DeleteTablet for tablet 900be14c60dc4be0adc22c2bcee4dd81 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:45 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:45.207930 31515 tablet_service.cc:1514] Processing DeleteTablet for tablet 900be14c60dc4be0adc22c2bcee4dd81 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:45 UTC) from {username='slave'} at 127.0.0.1:45194
I20240502 21:26:45.208189 2329 tablet_replica.cc:330] T 900be14c60dc4be0adc22c2bcee4dd81 P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:45.208390 2329 raft_consensus.cc:2229] T 900be14c60dc4be0adc22c2bcee4dd81 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:45.208380 31775 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a0d1c43a0794c2884dfef498f046f6d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:45 UTC) from {username='slave'} at 127.0.0.1:51596
I20240502 21:26:45.208375 31906 tablet_service.cc:1514] Processing DeleteTablet for tablet 900be14c60dc4be0adc22c2bcee4dd81 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:45 UTC) from {username='slave'} at 127.0.0.1:47824
I20240502 21:26:45.208611 2602 tablet_replica.cc:330] T 900be14c60dc4be0adc22c2bcee4dd81 P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:45.208649 2601 tablet_replica.cc:330] T 9a0d1c43a0794c2884dfef498f046f6d P 08d7a0680d244b55b93d17f9ab57b84c: stopping tablet replica
I20240502 21:26:45.208667 2328 tablet_replica.cc:330] T 900be14c60dc4be0adc22c2bcee4dd81 P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:26:45.208650 2329 raft_consensus.cc:2258] T 900be14c60dc4be0adc22c2bcee4dd81 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:45.208828 2602 raft_consensus.cc:2229] T 900be14c60dc4be0adc22c2bcee4dd81 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:45.208832 2328 raft_consensus.cc:2229] T 900be14c60dc4be0adc22c2bcee4dd81 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:45.208860 2601 raft_consensus.cc:2229] T 9a0d1c43a0794c2884dfef498f046f6d P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:45.209075 2602 raft_consensus.cc:2258] T 900be14c60dc4be0adc22c2bcee4dd81 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:45.209223 2601 raft_consensus.cc:2258] T 9a0d1c43a0794c2884dfef498f046f6d P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:45.209221 2328 raft_consensus.cc:2258] T 900be14c60dc4be0adc22c2bcee4dd81 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:45.209395 2329 ts_tablet_manager.cc:1875] T 900be14c60dc4be0adc22c2bcee4dd81 P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:45.209762 2602 ts_tablet_manager.cc:1875] T 900be14c60dc4be0adc22c2bcee4dd81 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:45.209843 2601 ts_tablet_manager.cc:1875] T 9a0d1c43a0794c2884dfef498f046f6d P 08d7a0680d244b55b93d17f9ab57b84c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:45.209915 2328 ts_tablet_manager.cc:1875] T 900be14c60dc4be0adc22c2bcee4dd81 P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:45.212343 31394 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:26:45.215107 31393 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 0af9a3f71bcc47bd9290d4d75f65ca24
I20240502 21:26:45.215628 31906 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a0d1c43a0794c2884dfef498f046f6d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:45 UTC) from {username='slave'} at 127.0.0.1:47824
I20240502 21:26:45.216200 31515 tablet_service.cc:1514] Processing DeleteTablet for tablet 0af9a3f71bcc47bd9290d4d75f65ca24 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:45 UTC) from {username='slave'} at 127.0.0.1:45194
I20240502 21:26:45.216560 31647 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a0d1c43a0794c2884dfef498f046f6d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:45 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:45.216660 31645 tablet_service.cc:1514] Processing DeleteTablet for tablet 0af9a3f71bcc47bd9290d4d75f65ca24 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:45 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:45.217267 2328 ts_tablet_manager.cc:1888] T 900be14c60dc4be0adc22c2bcee4dd81 P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:45.217515 31906 tablet_service.cc:1514] Processing DeleteTablet for tablet 0af9a3f71bcc47bd9290d4d75f65ca24 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:45 UTC) from {username='slave'} at 127.0.0.1:47824
I20240502 21:26:45.217670 2601 ts_tablet_manager.cc:1888] T 9a0d1c43a0794c2884dfef498f046f6d P 08d7a0680d244b55b93d17f9ab57b84c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:45.217396 2328 log.cc:1196] T 900be14c60dc4be0adc22c2bcee4dd81 P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/900be14c60dc4be0adc22c2bcee4dd81
I20240502 21:26:45.217830 2601 log.cc:1196] T 9a0d1c43a0794c2884dfef498f046f6d P 08d7a0680d244b55b93d17f9ab57b84c: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/wals/9a0d1c43a0794c2884dfef498f046f6d
I20240502 21:26:45.218230 2328 ts_tablet_manager.cc:1909] T 900be14c60dc4be0adc22c2bcee4dd81 P 2d50f33557944bb480ea32199d9ee4f1: Deleting consensus metadata
I20240502 21:26:45.218385 2601 ts_tablet_manager.cc:1909] T 9a0d1c43a0794c2884dfef498f046f6d P 08d7a0680d244b55b93d17f9ab57b84c: Deleting consensus metadata
I20240502 21:26:45.219389 2328 tablet_replica.cc:330] T 9a0d1c43a0794c2884dfef498f046f6d P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:26:45.219472 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet 900be14c60dc4be0adc22c2bcee4dd81 (table rebalance_test_table_10 [id=ff1e1957a03340f382098859ea2c90a8]) successfully deleted
I20240502 21:26:45.219439 31381 catalog_manager.cc:4830] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): tablet 9a0d1c43a0794c2884dfef498f046f6d (table rebalance_test_table_10 [id=ff1e1957a03340f382098859ea2c90a8]) successfully deleted
I20240502 21:26:45.219619 2328 raft_consensus.cc:2229] T 9a0d1c43a0794c2884dfef498f046f6d P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:45.219831 2328 raft_consensus.cc:2258] T 9a0d1c43a0794c2884dfef498f046f6d P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:45.220336 2329 ts_tablet_manager.cc:1888] T 900be14c60dc4be0adc22c2bcee4dd81 P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:45.220479 2329 log.cc:1196] T 900be14c60dc4be0adc22c2bcee4dd81 P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/900be14c60dc4be0adc22c2bcee4dd81
I20240502 21:26:45.220593 2328 ts_tablet_manager.cc:1875] T 9a0d1c43a0794c2884dfef498f046f6d P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:45.221606 2329 ts_tablet_manager.cc:1909] T 900be14c60dc4be0adc22c2bcee4dd81 P a44f41b0b6644fb6aa29476aed8c964f: Deleting consensus metadata
I20240502 21:26:45.222294 1834 raft_consensus.cc:1261] T bb7211c63ddd450bb260a32dfb6110b5 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:45.222294 31924 raft_consensus.cc:1261] T bb7211c63ddd450bb260a32dfb6110b5 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:45.222853 2631 consensus_queue.cc:1035] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:45.223114 2303 consensus_queue.cc:1035] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:45.223282 2329 tablet_replica.cc:330] T 0af9a3f71bcc47bd9290d4d75f65ca24 P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:45.224481 31666 consensus_queue.cc:237] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } }
I20240502 21:26:45.225642 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet 900be14c60dc4be0adc22c2bcee4dd81 (table rebalance_test_table_10 [id=ff1e1957a03340f382098859ea2c90a8]) successfully deleted
W20240502 21:26:45.226778 31603 consensus_peers.cc:485] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): Couldn't send request to peer 2d50f33557944bb480ea32199d9ee4f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49acc5301b9849e9b6795179acbc59a8. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:45.227152 2228 raft_consensus.cc:1261] T 49acc5301b9849e9b6795179acbc59a8 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:45.227926 2608 consensus_queue.cc:1035] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:45.227988 2097 raft_consensus.cc:1261] T 49acc5301b9849e9b6795179acbc59a8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:45.228541 2608 consensus_queue.cc:1035] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:45.230773 2329 raft_consensus.cc:2229] T 0af9a3f71bcc47bd9290d4d75f65ca24 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:45.231097 2329 raft_consensus.cc:2258] T 0af9a3f71bcc47bd9290d4d75f65ca24 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:45.231688 2608 raft_consensus.cc:2932] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } } }
I20240502 21:26:45.232524 2329 ts_tablet_manager.cc:1875] T 0af9a3f71bcc47bd9290d4d75f65ca24 P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:45.235901 2228 raft_consensus.cc:2932] T 49acc5301b9849e9b6795179acbc59a8 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } } }
I20240502 21:26:45.236809 2602 ts_tablet_manager.cc:1888] T 900be14c60dc4be0adc22c2bcee4dd81 P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:45.236945 2602 log.cc:1196] T 900be14c60dc4be0adc22c2bcee4dd81 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/900be14c60dc4be0adc22c2bcee4dd81
I20240502 21:26:45.237370 2602 ts_tablet_manager.cc:1909] T 900be14c60dc4be0adc22c2bcee4dd81 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:26:45.239076 31394 catalog_manager.cc:5484] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: config changed from index -1 to 2, NON_VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) added. New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:45.241010 2097 raft_consensus.cc:2932] T 49acc5301b9849e9b6795179acbc59a8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } } }
I20240502 21:26:45.242543 2329 ts_tablet_manager.cc:1888] T 0af9a3f71bcc47bd9290d4d75f65ca24 P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:45.242676 2329 log.cc:1196] T 0af9a3f71bcc47bd9290d4d75f65ca24 P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/0af9a3f71bcc47bd9290d4d75f65ca24
I20240502 21:26:45.243058 2329 ts_tablet_manager.cc:1909] T 0af9a3f71bcc47bd9290d4d75f65ca24 P a44f41b0b6644fb6aa29476aed8c964f: Deleting consensus metadata
I20240502 21:26:45.244058 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet 0af9a3f71bcc47bd9290d4d75f65ca24 (table rebalance_test_table_10 [id=ff1e1957a03340f382098859ea2c90a8]) successfully deleted
I20240502 21:26:45.248625 2328 ts_tablet_manager.cc:1888] T 9a0d1c43a0794c2884dfef498f046f6d P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:45.248770 2328 log.cc:1196] T 9a0d1c43a0794c2884dfef498f046f6d P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/9a0d1c43a0794c2884dfef498f046f6d
I20240502 21:26:45.249238 2328 ts_tablet_manager.cc:1909] T 9a0d1c43a0794c2884dfef498f046f6d P 2d50f33557944bb480ea32199d9ee4f1: Deleting consensus metadata
I20240502 21:26:45.250473 2328 tablet_replica.cc:330] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:26:45.250563 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet 9a0d1c43a0794c2884dfef498f046f6d (table rebalance_test_table_10 [id=ff1e1957a03340f382098859ea2c90a8]) successfully deleted
I20240502 21:26:45.250672 2328 raft_consensus.cc:2229] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:45.250877 2328 raft_consensus.cc:2258] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:45.251585 2328 ts_tablet_manager.cc:1875] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:45.258344 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet 900be14c60dc4be0adc22c2bcee4dd81 (table rebalance_test_table_10 [id=ff1e1957a03340f382098859ea2c90a8]) successfully deleted
I20240502 21:26:45.258750 2602 tablet_replica.cc:330] T 9a0d1c43a0794c2884dfef498f046f6d P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:45.258994 2602 raft_consensus.cc:2229] T 9a0d1c43a0794c2884dfef498f046f6d P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:45.259229 2602 raft_consensus.cc:2258] T 9a0d1c43a0794c2884dfef498f046f6d P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:45.262166 2602 ts_tablet_manager.cc:1875] T 9a0d1c43a0794c2884dfef498f046f6d P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:45.273378 2328 ts_tablet_manager.cc:1888] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:45.273496 2328 log.cc:1196] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/0af9a3f71bcc47bd9290d4d75f65ca24
I20240502 21:26:45.273855 2328 ts_tablet_manager.cc:1909] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 2d50f33557944bb480ea32199d9ee4f1: Deleting consensus metadata
I20240502 21:26:45.274833 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet 0af9a3f71bcc47bd9290d4d75f65ca24 (table rebalance_test_table_10 [id=ff1e1957a03340f382098859ea2c90a8]) successfully deleted
I20240502 21:26:45.275851 2602 ts_tablet_manager.cc:1888] T 9a0d1c43a0794c2884dfef498f046f6d P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:45.275986 2602 log.cc:1196] T 9a0d1c43a0794c2884dfef498f046f6d P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/9a0d1c43a0794c2884dfef498f046f6d
I20240502 21:26:45.276463 2602 ts_tablet_manager.cc:1909] T 9a0d1c43a0794c2884dfef498f046f6d P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:26:45.277201 31394 catalog_manager.cc:5576] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863) does not have the latest schema for tablet bb7211c63ddd450bb260a32dfb6110b5 (table rebalancer_dynamic_table [id=85a71ef99c954778a60e20dc5a6b3d15]). Expected version 1 got 0
I20240502 21:26:45.278290 31393 catalog_manager.cc:5576] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933) does not have the latest schema for tablet bb7211c63ddd450bb260a32dfb6110b5 (table rebalancer_dynamic_table [id=85a71ef99c954778a60e20dc5a6b3d15]). Expected version 1 got 0
I20240502 21:26:45.280529 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet 9a0d1c43a0794c2884dfef498f046f6d (table rebalance_test_table_10 [id=ff1e1957a03340f382098859ea2c90a8]) successfully deleted
I20240502 21:26:45.280879 2602 tablet_replica.cc:330] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:45.281173 2602 raft_consensus.cc:2229] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:45.281446 2602 raft_consensus.cc:2258] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:45.281076 2679 tablet.cc:1722] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:45.282275 2602 ts_tablet_manager.cc:1875] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:45.285058 2679 tablet.cc:1620] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f: MemRowSet was empty: no flush needed.
I20240502 21:26:45.286994 2680 tablet.cc:1722] T bb7211c63ddd450bb260a32dfb6110b5 P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:45.287168 31393 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=85a71ef99c954778a60e20dc5a6b3d15] alter complete (version 1)
I20240502 21:26:45.287508 2680 tablet.cc:1620] T bb7211c63ddd450bb260a32dfb6110b5 P 1e04157b98dd4fad98858c290b73c3db: MemRowSet was empty: no flush needed.
I20240502 21:26:45.288141 2681 tablet.cc:1722] T bb7211c63ddd450bb260a32dfb6110b5 P 2d50f33557944bb480ea32199d9ee4f1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:45.290578 31393 catalog_manager.cc:5576] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933) does not have the latest schema for tablet bb7211c63ddd450bb260a32dfb6110b5 (table rebalancer_dynamic_table [id=85a71ef99c954778a60e20dc5a6b3d15]). Expected version 1 got 0
I20240502 21:26:45.291874 2602 ts_tablet_manager.cc:1888] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:45.291997 2602 log.cc:1196] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/0af9a3f71bcc47bd9290d4d75f65ca24
I20240502 21:26:45.294091 31393 catalog_manager.cc:5576] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863) does not have the latest schema for tablet bb7211c63ddd450bb260a32dfb6110b5 (table rebalancer_dynamic_table [id=85a71ef99c954778a60e20dc5a6b3d15]). Expected version 1 got 0
I20240502 21:26:45.295624 2681 tablet.cc:1620] T bb7211c63ddd450bb260a32dfb6110b5 P 2d50f33557944bb480ea32199d9ee4f1: MemRowSet was empty: no flush needed.
I20240502 21:26:45.301271 2602 ts_tablet_manager.cc:1909] T 0af9a3f71bcc47bd9290d4d75f65ca24 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:26:45.303012 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet 0af9a3f71bcc47bd9290d4d75f65ca24 (table rebalance_test_table_10 [id=ff1e1957a03340f382098859ea2c90a8]) successfully deleted
I20240502 21:26:45.395309 31391 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:26:45.400831 2365 consensus_queue.cc:1035] T 579fc086d7524e488f8dc4526b89597b P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:45.412686 2679 tablet.cc:1722] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:45.413309 2679 tablet.cc:1620] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f: MemRowSet was empty: no flush needed.
I20240502 21:26:45.416611 2680 tablet.cc:1722] T bb7211c63ddd450bb260a32dfb6110b5 P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:45.417066 2680 tablet.cc:1620] T bb7211c63ddd450bb260a32dfb6110b5 P 1e04157b98dd4fad98858c290b73c3db: MemRowSet was empty: no flush needed.
I20240502 21:26:45.420747 2681 tablet.cc:1722] T bb7211c63ddd450bb260a32dfb6110b5 P 2d50f33557944bb480ea32199d9ee4f1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:45.421274 2681 tablet.cc:1620] T bb7211c63ddd450bb260a32dfb6110b5 P 2d50f33557944bb480ea32199d9ee4f1: MemRowSet was empty: no flush needed.
I20240502 21:26:45.429201 31395 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=85a71ef99c954778a60e20dc5a6b3d15] alter complete (version 2)
I20240502 21:26:45.443341 2365 consensus_queue.cc:1035] T 579fc086d7524e488f8dc4526b89597b P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:45.484328 31645 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:26:45.494287 1813 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:45.494746 31775 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 21:26:45.506736 1946 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:45.522742 2075 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:45.531841 31906 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:26:45.538134 31515 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:26:45.545925 31393 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:26:45.558233 2679 tablet.cc:1722] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:45.558952 2680 tablet.cc:1722] T bb7211c63ddd450bb260a32dfb6110b5 P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:45.562651 1884 consensus_queue.cc:1035] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:45.565317 1884 consensus_queue.cc:1035] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:45.568958 31393 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table_new_name [id=85a71ef99c954778a60e20dc5a6b3d15] alter complete (version 3)
I20240502 21:26:45.569566 2681 tablet.cc:1722] T bb7211c63ddd450bb260a32dfb6110b5 P 2d50f33557944bb480ea32199d9ee4f1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:45.578814 2429 consensus_queue.cc:1035] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:45.590943 2210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:45.600394 31393 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_extra_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:45.620234 1946 tablet_service.cc:1467] Processing CreateTablet for tablet 9bc57b8a0388416290997df544e5cc0b (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=86898baa76944823b3c5e8aba98d8772]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:45.620832 1946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bc57b8a0388416290997df544e5cc0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:45.622031 2210 tablet_service.cc:1467] Processing CreateTablet for tablet 577e7a586ad9459f8e77ce4b8f563c09 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=86898baa76944823b3c5e8aba98d8772]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:45.622431 2210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 577e7a586ad9459f8e77ce4b8f563c09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:45.625242 31515 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc6c7211d5f4f73b2a050b9ec78c6b1 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=86898baa76944823b3c5e8aba98d8772]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:45.625674 31515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc6c7211d5f4f73b2a050b9ec78c6b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:45.629212 2075 tablet_service.cc:1467] Processing CreateTablet for tablet 9bc57b8a0388416290997df544e5cc0b (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=86898baa76944823b3c5e8aba98d8772]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:45.629621 2075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bc57b8a0388416290997df544e5cc0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:45.632742 2077 tablet_service.cc:1467] Processing CreateTablet for tablet 577e7a586ad9459f8e77ce4b8f563c09 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=86898baa76944823b3c5e8aba98d8772]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:45.633028 2077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 577e7a586ad9459f8e77ce4b8f563c09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:45.636729 2076 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc6c7211d5f4f73b2a050b9ec78c6b1 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=86898baa76944823b3c5e8aba98d8772]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:45.637017 2076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc6c7211d5f4f73b2a050b9ec78c6b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:45.638178 1814 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc6c7211d5f4f73b2a050b9ec78c6b1 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=86898baa76944823b3c5e8aba98d8772]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:45.638604 1814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc6c7211d5f4f73b2a050b9ec78c6b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:45.638774 2725 tablet_bootstrap.cc:492] T 577e7a586ad9459f8e77ce4b8f563c09 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:45.641099 1813 tablet_service.cc:1467] Processing CreateTablet for tablet 9bc57b8a0388416290997df544e5cc0b (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=86898baa76944823b3c5e8aba98d8772]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:45.641477 1813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bc57b8a0388416290997df544e5cc0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:45.642377 2725 tablet_bootstrap.cc:654] T 577e7a586ad9459f8e77ce4b8f563c09 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:45.644721 1812 tablet_service.cc:1467] Processing CreateTablet for tablet 577e7a586ad9459f8e77ce4b8f563c09 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=86898baa76944823b3c5e8aba98d8772]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:45.645020 1812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 577e7a586ad9459f8e77ce4b8f563c09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:45.651726 1966 raft_consensus.cc:3037] T 1022c4c6ee8642ca95250933a940fe7a P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:45.653759 2726 ts_tablet_manager.cc:902] T bb64e0d78397429d959535f570658594 P 9dec645b473f40578e726f3e50e219bf: Initiating tablet copy from peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:45.655838 2429 consensus_queue.cc:1035] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:45.686933 31394 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:45.687139 31394 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:45.694206 31394 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet bb7211c63ddd450bb260a32dfb6110b5
I20240502 21:26:45.694845 2730 ts_tablet_manager.cc:902] T f02f83ec0a494da5b66241d9b379a03f P a44f41b0b6644fb6aa29476aed8c964f: Initiating tablet copy from peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:26:45.695655 31906 tablet_service.cc:1514] Processing DeleteTablet for tablet bb7211c63ddd450bb260a32dfb6110b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:45 UTC) from {username='slave'} at 127.0.0.1:47824
I20240502 21:26:45.695991 31516 tablet_service.cc:1514] Processing DeleteTablet for tablet bb7211c63ddd450bb260a32dfb6110b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:45 UTC) from {username='slave'} at 127.0.0.1:45194
I20240502 21:26:45.696231 2329 tablet_replica.cc:330] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:45.696226 2328 tablet_replica.cc:330] T bb7211c63ddd450bb260a32dfb6110b5 P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:26:45.696422 2329 raft_consensus.cc:2229] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:45.696501 2328 raft_consensus.cc:2229] T bb7211c63ddd450bb260a32dfb6110b5 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:45.696682 2328 raft_consensus.cc:2258] T bb7211c63ddd450bb260a32dfb6110b5 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:45.696767 2329 raft_consensus.cc:2258] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:45.697573 2329 ts_tablet_manager.cc:1875] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:45.698010 2328 ts_tablet_manager.cc:1875] T bb7211c63ddd450bb260a32dfb6110b5 P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:45.703244 2329 ts_tablet_manager.cc:1888] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:45.703382 2329 log.cc:1196] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/bb7211c63ddd450bb260a32dfb6110b5
I20240502 21:26:45.703857 2329 ts_tablet_manager.cc:1909] T bb7211c63ddd450bb260a32dfb6110b5 P a44f41b0b6644fb6aa29476aed8c964f: Deleting consensus metadata
I20240502 21:26:45.707762 2328 ts_tablet_manager.cc:1888] T bb7211c63ddd450bb260a32dfb6110b5 P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:45.708276 2328 log.cc:1196] T bb7211c63ddd450bb260a32dfb6110b5 P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/bb7211c63ddd450bb260a32dfb6110b5
I20240502 21:26:45.708752 2328 ts_tablet_manager.cc:1909] T bb7211c63ddd450bb260a32dfb6110b5 P 2d50f33557944bb480ea32199d9ee4f1: Deleting consensus metadata
I20240502 21:26:45.709913 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet bb7211c63ddd450bb260a32dfb6110b5 (table rebalancer_dynamic_table_new_name [id=85a71ef99c954778a60e20dc5a6b3d15]) successfully deleted
I20240502 21:26:45.711002 1811 tablet_service.cc:1514] Processing DeleteTablet for tablet bb7211c63ddd450bb260a32dfb6110b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:45 UTC) from {username='slave'} at 127.0.0.1:47510
I20240502 21:26:45.717031 31394 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240502 21:26:45.718643 31394 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240502 21:26:45.719014 2735 tablet_bootstrap.cc:492] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:45.723443 31394 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet dc734c0b3a094207856d5cd56c0e73b8
I20240502 21:26:45.724341 31775 tablet_service.cc:1514] Processing DeleteTablet for tablet dc734c0b3a094207856d5cd56c0e73b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:45 UTC) from {username='slave'} at 127.0.0.1:51596
I20240502 21:26:45.726447 31516 tablet_service.cc:1514] Processing DeleteTablet for tablet dc734c0b3a094207856d5cd56c0e73b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:45 UTC) from {username='slave'} at 127.0.0.1:45194
I20240502 21:26:45.727974 31645 tablet_service.cc:1514] Processing DeleteTablet for tablet dc734c0b3a094207856d5cd56c0e73b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:45 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:45.728279 2602 tablet_replica.cc:330] T dc734c0b3a094207856d5cd56c0e73b8 P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:45.728487 2602 raft_consensus.cc:2229] T dc734c0b3a094207856d5cd56c0e73b8 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:45.728071 31394 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet cc8ef061e4224c8d94a06e47dd51a5d1
I20240502 21:26:45.728755 2602 raft_consensus.cc:2258] T dc734c0b3a094207856d5cd56c0e73b8 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:45.729333 31906 tablet_service.cc:1514] Processing DeleteTablet for tablet cc8ef061e4224c8d94a06e47dd51a5d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:45 UTC) from {username='slave'} at 127.0.0.1:47824
I20240502 21:26:45.729569 2602 ts_tablet_manager.cc:1875] T dc734c0b3a094207856d5cd56c0e73b8 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:45.729652 2328 tablet_replica.cc:330] T cc8ef061e4224c8d94a06e47dd51a5d1 P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:26:45.729823 2328 raft_consensus.cc:2229] T cc8ef061e4224c8d94a06e47dd51a5d1 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:45.730033 2328 raft_consensus.cc:2258] T cc8ef061e4224c8d94a06e47dd51a5d1 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:45.730396 31645 tablet_service.cc:1514] Processing DeleteTablet for tablet cc8ef061e4224c8d94a06e47dd51a5d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:45 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:45.730477 31394 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 4 replicas of tablet bb64e0d78397429d959535f570658594
I20240502 21:26:45.730669 2328 ts_tablet_manager.cc:1875] T cc8ef061e4224c8d94a06e47dd51a5d1 P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:45.731076 31774 tablet_service.cc:1514] Processing DeleteTablet for tablet bb64e0d78397429d959535f570658594 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:45 UTC) from {username='slave'} at 127.0.0.1:51596
I20240502 21:26:45.731920 31515 tablet_service.cc:1514] Processing DeleteTablet for tablet cc8ef061e4224c8d94a06e47dd51a5d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:45 UTC) from {username='slave'} at 127.0.0.1:45194
I20240502 21:26:45.732244 31645 tablet_service.cc:1514] Processing DeleteTablet for tablet bb64e0d78397429d959535f570658594 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:45 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:45.732872 1945 tablet_service.cc:1514] Processing DeleteTablet for tablet bb64e0d78397429d959535f570658594 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:45 UTC) from {username='slave'} at 127.0.0.1:33002
I20240502 21:26:45.737951 31906 tablet_service.cc:1514] Processing DeleteTablet for tablet bb64e0d78397429d959535f570658594 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:45 UTC) from {username='slave'} at 127.0.0.1:47824
I20240502 21:26:45.738049 2602 ts_tablet_manager.cc:1888] T dc734c0b3a094207856d5cd56c0e73b8 P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:45.738215 2602 log.cc:1196] T dc734c0b3a094207856d5cd56c0e73b8 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/dc734c0b3a094207856d5cd56c0e73b8
I20240502 21:26:45.738605 2602 ts_tablet_manager.cc:1909] T dc734c0b3a094207856d5cd56c0e73b8 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:26:45.740406 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet dc734c0b3a094207856d5cd56c0e73b8 (table rebalance_test_table_12 [id=2ef1b694f2424f519337e0d5c334e4d5]) successfully deleted
I20240502 21:26:45.740780 2602 tablet_replica.cc:330] T cc8ef061e4224c8d94a06e47dd51a5d1 P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:45.741003 2602 raft_consensus.cc:2229] T cc8ef061e4224c8d94a06e47dd51a5d1 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:45.741297 2602 raft_consensus.cc:2258] T cc8ef061e4224c8d94a06e47dd51a5d1 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:45.741344 2328 ts_tablet_manager.cc:1888] T cc8ef061e4224c8d94a06e47dd51a5d1 P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:45.741649 2328 log.cc:1196] T cc8ef061e4224c8d94a06e47dd51a5d1 P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/cc8ef061e4224c8d94a06e47dd51a5d1
I20240502 21:26:45.741997 2602 ts_tablet_manager.cc:1875] T cc8ef061e4224c8d94a06e47dd51a5d1 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:45.742259 2328 ts_tablet_manager.cc:1909] T cc8ef061e4224c8d94a06e47dd51a5d1 P 2d50f33557944bb480ea32199d9ee4f1: Deleting consensus metadata
I20240502 21:26:45.743556 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet cc8ef061e4224c8d94a06e47dd51a5d1 (table rebalance_test_table_12 [id=2ef1b694f2424f519337e0d5c334e4d5]) successfully deleted
I20240502 21:26:45.745643 2328 tablet_replica.cc:330] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:26:45.745879 2328 raft_consensus.cc:2229] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:45.746328 2328 raft_consensus.cc:2258] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:45.747294 2328 ts_tablet_manager.cc:1875] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 21:26:45.747393 31379 catalog_manager.cc:4797] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): delete failed for tablet bb64e0d78397429d959535f570658594 because the tablet was not found. No further retry: Not found: Tablet not found: bb64e0d78397429d959535f570658594
I20240502 21:26:45.750459 2735 tablet_bootstrap.cc:654] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:45.755318 2739 tablet_replica.cc:330] T dc734c0b3a094207856d5cd56c0e73b8 P 08d7a0680d244b55b93d17f9ab57b84c: stopping tablet replica
I20240502 21:26:45.755674 2602 ts_tablet_manager.cc:1888] T cc8ef061e4224c8d94a06e47dd51a5d1 P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:45.755663 2739 raft_consensus.cc:2229] T dc734c0b3a094207856d5cd56c0e73b8 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:45.755806 2602 log.cc:1196] T cc8ef061e4224c8d94a06e47dd51a5d1 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/cc8ef061e4224c8d94a06e47dd51a5d1
I20240502 21:26:45.756055 2739 raft_consensus.cc:2258] T dc734c0b3a094207856d5cd56c0e73b8 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:45.756209 2602 ts_tablet_manager.cc:1909] T cc8ef061e4224c8d94a06e47dd51a5d1 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:26:45.758569 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet bb7211c63ddd450bb260a32dfb6110b5 (table rebalancer_dynamic_table_new_name [id=85a71ef99c954778a60e20dc5a6b3d15]) successfully deleted
I20240502 21:26:45.758935 2329 tablet_replica.cc:330] T dc734c0b3a094207856d5cd56c0e73b8 P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:45.759868 2739 ts_tablet_manager.cc:1875] T dc734c0b3a094207856d5cd56c0e73b8 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:45.760047 2329 raft_consensus.cc:2229] T dc734c0b3a094207856d5cd56c0e73b8 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:45.760949 2329 raft_consensus.cc:2258] T dc734c0b3a094207856d5cd56c0e73b8 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:45.762243 2328 ts_tablet_manager.cc:1888] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:45.762378 2328 log.cc:1196] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/bb64e0d78397429d959535f570658594
I20240502 21:26:45.762766 2328 ts_tablet_manager.cc:1909] T bb64e0d78397429d959535f570658594 P 2d50f33557944bb480ea32199d9ee4f1: Deleting consensus metadata
I20240502 21:26:45.763037 2329 ts_tablet_manager.cc:1875] T dc734c0b3a094207856d5cd56c0e73b8 P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:45.763826 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet bb64e0d78397429d959535f570658594 (table rebalance_test_table_12 [id=2ef1b694f2424f519337e0d5c334e4d5]) successfully deleted
I20240502 21:26:45.764601 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet cc8ef061e4224c8d94a06e47dd51a5d1 (table rebalance_test_table_12 [id=2ef1b694f2424f519337e0d5c334e4d5]) successfully deleted
I20240502 21:26:45.764837 2602 tablet_replica.cc:330] T bb64e0d78397429d959535f570658594 P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:45.765023 2602 raft_consensus.cc:2229] T bb64e0d78397429d959535f570658594 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:45.765794 2602 raft_consensus.cc:2258] T bb64e0d78397429d959535f570658594 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:45.770057 2729 tablet_bootstrap.cc:492] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:45.780663 2730 tablet_copy_client.cc:308] T f02f83ec0a494da5b66241d9b379a03f P a44f41b0b6644fb6aa29476aed8c964f: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.134:43753
I20240502 21:26:45.781512 2729 tablet_bootstrap.cc:654] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:45.781553 2602 ts_tablet_manager.cc:1875] T bb64e0d78397429d959535f570658594 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:45.787705 2602 ts_tablet_manager.cc:1888] T bb64e0d78397429d959535f570658594 P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:45.792037 2735 tablet_bootstrap.cc:492] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:45.792243 2735 ts_tablet_manager.cc:1367] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.073s user 0.005s sys 0.000s
I20240502 21:26:45.793203 2735 raft_consensus.cc:348] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:45.793444 2602 log.cc:1196] T bb64e0d78397429d959535f570658594 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/bb64e0d78397429d959535f570658594
I20240502 21:26:45.793663 2735 raft_consensus.cc:374] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:45.793831 2735 raft_consensus.cc:729] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:45.794148 2735 consensus_queue.cc:260] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:45.794956 2735 ts_tablet_manager.cc:1398] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:26:45.797219 2602 ts_tablet_manager.cc:1909] T bb64e0d78397429d959535f570658594 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:26:45.797247 31394 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:26:45.799302 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet bb64e0d78397429d959535f570658594 (table rebalance_test_table_12 [id=2ef1b694f2424f519337e0d5c334e4d5]) successfully deleted
I20240502 21:26:45.799525 2739 ts_tablet_manager.cc:1888] T dc734c0b3a094207856d5cd56c0e73b8 P 08d7a0680d244b55b93d17f9ab57b84c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:45.800257 2739 log.cc:1196] T dc734c0b3a094207856d5cd56c0e73b8 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/wals/dc734c0b3a094207856d5cd56c0e73b8
I20240502 21:26:45.801172 2739 ts_tablet_manager.cc:1909] T dc734c0b3a094207856d5cd56c0e73b8 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting consensus metadata
I20240502 21:26:45.807909 1986 tablet_copy_service.cc:140] P 9dec645b473f40578e726f3e50e219bf: Received BeginTabletCopySession request for tablet f02f83ec0a494da5b66241d9b379a03f from peer a44f41b0b6644fb6aa29476aed8c964f ({username='slave'} at 127.24.226.129:41327)
I20240502 21:26:45.808135 1986 tablet_copy_service.cc:161] P 9dec645b473f40578e726f3e50e219bf: Beginning new tablet copy session on tablet f02f83ec0a494da5b66241d9b379a03f from peer a44f41b0b6644fb6aa29476aed8c964f at {username='slave'} at 127.24.226.129:41327: session id = a44f41b0b6644fb6aa29476aed8c964f-f02f83ec0a494da5b66241d9b379a03f
I20240502 21:26:45.809007 2736 tablet_bootstrap.cc:492] T 9bc57b8a0388416290997df544e5cc0b P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:45.810883 2736 tablet_bootstrap.cc:654] T 9bc57b8a0388416290997df544e5cc0b P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:45.811426 31515 tablet_service.cc:1467] Processing CreateTablet for tablet 594137cda8be4137beae728786a7731d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8123e6cbc25b44e1899a876d634dfaa5]), partition=
I20240502 21:26:45.811445 31906 tablet_service.cc:1467] Processing CreateTablet for tablet 594137cda8be4137beae728786a7731d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8123e6cbc25b44e1899a876d634dfaa5]), partition=
I20240502 21:26:45.811733 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 594137cda8be4137beae728786a7731d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:45.811759 31515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 594137cda8be4137beae728786a7731d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:45.811944 1986 tablet_copy_source_session.cc:205] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:45.813288 2739 tablet_replica.cc:330] T bb64e0d78397429d959535f570658594 P 08d7a0680d244b55b93d17f9ab57b84c: stopping tablet replica
I20240502 21:26:45.816082 31381 catalog_manager.cc:4830] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): tablet dc734c0b3a094207856d5cd56c0e73b8 (table rebalance_test_table_12 [id=2ef1b694f2424f519337e0d5c334e4d5]) successfully deleted
I20240502 21:26:45.816608 1946 tablet_service.cc:1467] Processing CreateTablet for tablet 594137cda8be4137beae728786a7731d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8123e6cbc25b44e1899a876d634dfaa5]), partition=
I20240502 21:26:45.816915 1946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 594137cda8be4137beae728786a7731d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:45.826719 2747 ts_tablet_manager.cc:902] T 49acc5301b9849e9b6795179acbc59a8 P 2d50f33557944bb480ea32199d9ee4f1: Initiating tablet copy from peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:45.827503 2732 tablet_bootstrap.cc:492] T 9bc57b8a0388416290997df544e5cc0b P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:45.829164 2732 tablet_bootstrap.cc:654] T 9bc57b8a0388416290997df544e5cc0b P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:45.832406 2748 tablet_bootstrap.cc:492] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:45.832777 2730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f02f83ec0a494da5b66241d9b379a03f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:45.834085 2748 tablet_bootstrap.cc:654] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:45.835963 2738 tablet_replica.cc:330] T bb7211c63ddd450bb260a32dfb6110b5 P 1e04157b98dd4fad98858c290b73c3db: stopping tablet replica
I20240502 21:26:45.837200 2738 raft_consensus.cc:2229] T bb7211c63ddd450bb260a32dfb6110b5 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:45.837564 2738 raft_consensus.cc:2258] T bb7211c63ddd450bb260a32dfb6110b5 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:45.838619 2724 ts_tablet_manager.cc:902] T 182b3f15c06c4c07ade0a8854d3ecafd P 1e04157b98dd4fad98858c290b73c3db: Initiating tablet copy from peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:45.840086 2729 tablet_bootstrap.cc:492] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:45.840263 2729 ts_tablet_manager.cc:1367] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.070s user 0.005s sys 0.000s
I20240502 21:26:45.840976 2729 raft_consensus.cc:348] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:45.841208 2729 raft_consensus.cc:374] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:45.841310 2729 raft_consensus.cc:729] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:45.841564 2729 consensus_queue.cc:260] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:45.849125 2729 ts_tablet_manager.cc:1398] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 21:26:45.850932 2736 tablet_bootstrap.cc:492] T 9bc57b8a0388416290997df544e5cc0b P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:26:45.852140 2736 ts_tablet_manager.cc:1367] T 9bc57b8a0388416290997df544e5cc0b P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.043s user 0.002s sys 0.003s
I20240502 21:26:45.853796 2329 ts_tablet_manager.cc:1888] T dc734c0b3a094207856d5cd56c0e73b8 P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:45.854043 2329 log.cc:1196] T dc734c0b3a094207856d5cd56c0e73b8 P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/dc734c0b3a094207856d5cd56c0e73b8
I20240502 21:26:45.854672 2329 ts_tablet_manager.cc:1909] T dc734c0b3a094207856d5cd56c0e73b8 P a44f41b0b6644fb6aa29476aed8c964f: Deleting consensus metadata
I20240502 21:26:45.855851 2329 tablet_replica.cc:330] T cc8ef061e4224c8d94a06e47dd51a5d1 P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:45.856014 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet dc734c0b3a094207856d5cd56c0e73b8 (table rebalance_test_table_12 [id=2ef1b694f2424f519337e0d5c334e4d5]) successfully deleted
I20240502 21:26:45.856086 2329 raft_consensus.cc:2229] T cc8ef061e4224c8d94a06e47dd51a5d1 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:45.858423 2748 tablet_bootstrap.cc:492] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:45.859486 2739 raft_consensus.cc:2229] T bb64e0d78397429d959535f570658594 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:45.859803 2736 raft_consensus.cc:348] T 9bc57b8a0388416290997df544e5cc0b P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:45.860633 2735 tablet_bootstrap.cc:492] T 594137cda8be4137beae728786a7731d P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:45.860750 2736 raft_consensus.cc:374] T 9bc57b8a0388416290997df544e5cc0b P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:45.860867 2736 raft_consensus.cc:729] T 9bc57b8a0388416290997df544e5cc0b P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:26:45.861122 2748 ts_tablet_manager.cc:1367] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.003s
I20240502 21:26:45.862497 2736 consensus_queue.cc:260] T 9bc57b8a0388416290997df544e5cc0b P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:45.863396 2748 raft_consensus.cc:348] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:45.865268 2739 raft_consensus.cc:2258] T bb64e0d78397429d959535f570658594 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:45.867023 2739 ts_tablet_manager.cc:1875] T bb64e0d78397429d959535f570658594 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:45.868556 2735 tablet_bootstrap.cc:654] T 594137cda8be4137beae728786a7731d P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:45.868918 2736 ts_tablet_manager.cc:1398] T 9bc57b8a0388416290997df544e5cc0b P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20240502 21:26:45.870216 2736 tablet_bootstrap.cc:492] T 594137cda8be4137beae728786a7731d P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:45.870496 2748 raft_consensus.cc:374] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:45.871862 2748 raft_consensus.cc:729] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:45.873208 2748 consensus_queue.cc:260] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:45.875923 2748 ts_tablet_manager.cc:1398] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20240502 21:26:45.877090 2736 tablet_bootstrap.cc:654] T 594137cda8be4137beae728786a7731d P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:45.878798 2329 raft_consensus.cc:2258] T cc8ef061e4224c8d94a06e47dd51a5d1 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:45.879094 2730 tablet_copy_client.cc:783] T f02f83ec0a494da5b66241d9b379a03f P a44f41b0b6644fb6aa29476aed8c964f: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:45.885138 2730 tablet_copy_client.cc:647] T f02f83ec0a494da5b66241d9b379a03f P a44f41b0b6644fb6aa29476aed8c964f: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:45.886656 2725 tablet_bootstrap.cc:492] T 577e7a586ad9459f8e77ce4b8f563c09 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:45.886795 2725 ts_tablet_manager.cc:1367] T 577e7a586ad9459f8e77ce4b8f563c09 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.255s user 0.061s sys 0.080s
I20240502 21:26:45.887559 2725 raft_consensus.cc:348] T 577e7a586ad9459f8e77ce4b8f563c09 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:45.887780 2725 raft_consensus.cc:374] T 577e7a586ad9459f8e77ce4b8f563c09 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:45.887877 2725 raft_consensus.cc:729] T 577e7a586ad9459f8e77ce4b8f563c09 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:45.887895 2515 raft_consensus.cc:482] T 9bc57b8a0388416290997df544e5cc0b P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:45.888031 2515 raft_consensus.cc:504] T 9bc57b8a0388416290997df544e5cc0b P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:45.888108 2725 consensus_queue.cc:260] T 577e7a586ad9459f8e77ce4b8f563c09 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:45.874045 2359 raft_consensus.cc:482] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:45.888751 2725 ts_tablet_manager.cc:1398] T 577e7a586ad9459f8e77ce4b8f563c09 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:45.888818 2515 leader_election.cc:288] T 9bc57b8a0388416290997df544e5cc0b P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:26:45.888746 2359 raft_consensus.cc:504] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:45.889427 2359 leader_election.cc:288] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:45.889554 2329 ts_tablet_manager.cc:1875] T cc8ef061e4224c8d94a06e47dd51a5d1 P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:45.890381 2228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577e7a586ad9459f8e77ce4b8f563c09" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:26:45.890457 2729 tablet_bootstrap.cc:492] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:45.890594 2228 raft_consensus.cc:2454] T 577e7a586ad9459f8e77ce4b8f563c09 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 0.
I20240502 21:26:45.891009 1769 leader_election.cc:302] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:45.891400 2359 raft_consensus.cc:2781] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:45.891542 2359 raft_consensus.cc:482] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:45.891644 2359 raft_consensus.cc:3037] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:45.892359 2729 tablet_bootstrap.cc:654] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:45.892976 2095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bc57b8a0388416290997df544e5cc0b" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:45.882316 2738 ts_tablet_manager.cc:1875] T bb7211c63ddd450bb260a32dfb6110b5 P 1e04157b98dd4fad98858c290b73c3db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:45.895128 2359 raft_consensus.cc:504] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:45.895619 2097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577e7a586ad9459f8e77ce4b8f563c09" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:45.896054 2095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577e7a586ad9459f8e77ce4b8f563c09" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
W20240502 21:26:45.896196 1769 leader_election.cc:341] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:45.896639 2359 leader_election.cc:288] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Requested vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
W20240502 21:26:45.896692 1769 leader_election.cc:341] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:45.898294 2228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577e7a586ad9459f8e77ce4b8f563c09" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:26:45.898548 2228 raft_consensus.cc:3037] T 577e7a586ad9459f8e77ce4b8f563c09 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:45.901438 2228 raft_consensus.cc:2454] T 577e7a586ad9459f8e77ce4b8f563c09 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:26:45.901952 1769 leader_election.cc:302] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, d1406e92aec84ac585304e57bd37a40d; no voters: 1ade0354c44c4c37ae480c7a94b11e1b
I20240502 21:26:45.902606 2359 raft_consensus.cc:2781] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:45.904172 2739 ts_tablet_manager.cc:1888] T bb64e0d78397429d959535f570658594 P 08d7a0680d244b55b93d17f9ab57b84c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:45.909641 2329 ts_tablet_manager.cc:1888] T cc8ef061e4224c8d94a06e47dd51a5d1 P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:45.909847 2329 log.cc:1196] T cc8ef061e4224c8d94a06e47dd51a5d1 P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/cc8ef061e4224c8d94a06e47dd51a5d1
I20240502 21:26:45.910394 2329 ts_tablet_manager.cc:1909] T cc8ef061e4224c8d94a06e47dd51a5d1 P a44f41b0b6644fb6aa29476aed8c964f: Deleting consensus metadata
I20240502 21:26:45.910907 2739 log.cc:1196] T bb64e0d78397429d959535f570658594 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/wals/bb64e0d78397429d959535f570658594
I20240502 21:26:45.911533 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet cc8ef061e4224c8d94a06e47dd51a5d1 (table rebalance_test_table_12 [id=2ef1b694f2424f519337e0d5c334e4d5]) successfully deleted
I20240502 21:26:45.912387 2739 ts_tablet_manager.cc:1909] T bb64e0d78397429d959535f570658594 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting consensus metadata
I20240502 21:26:45.913553 2359 raft_consensus.cc:686] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Becoming Leader. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Running, Role: LEADER
I20240502 21:26:45.913923 2359 consensus_queue.cc:237] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:45.917235 31381 catalog_manager.cc:4830] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): tablet bb64e0d78397429d959535f570658594 (table rebalance_test_table_12 [id=2ef1b694f2424f519337e0d5c334e4d5]) successfully deleted
I20240502 21:26:45.917619 2735 tablet_bootstrap.cc:492] T 594137cda8be4137beae728786a7731d P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:45.917834 2735 ts_tablet_manager.cc:1367] T 594137cda8be4137beae728786a7731d P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.057s user 0.001s sys 0.012s
I20240502 21:26:45.918682 2735 raft_consensus.cc:348] T 594137cda8be4137beae728786a7731d P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:45.918980 2735 raft_consensus.cc:374] T 594137cda8be4137beae728786a7731d P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:45.919111 2735 raft_consensus.cc:729] T 594137cda8be4137beae728786a7731d P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
W20240502 21:26:45.919382 1901 leader_election.cc:341] T 9bc57b8a0388416290997df544e5cc0b P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:45.919380 2735 consensus_queue.cc:260] T 594137cda8be4137beae728786a7731d P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:45.919916 2730 tablet_copy_client.cc:523] T f02f83ec0a494da5b66241d9b379a03f P a44f41b0b6644fb6aa29476aed8c964f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:45.920086 2735 ts_tablet_manager.cc:1398] T 594137cda8be4137beae728786a7731d P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:45.927304 31389 catalog_manager.cc:5484] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e04157b98dd4fad98858c290b73c3db (127.24.226.133). New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:45.929116 2736 tablet_bootstrap.cc:492] T 594137cda8be4137beae728786a7731d P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:26:45.929342 2736 ts_tablet_manager.cc:1367] T 594137cda8be4137beae728786a7731d P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.059s user 0.004s sys 0.001s
I20240502 21:26:45.930425 2736 raft_consensus.cc:348] T 594137cda8be4137beae728786a7731d P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:45.930733 2736 raft_consensus.cc:374] T 594137cda8be4137beae728786a7731d P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:45.930867 2736 raft_consensus.cc:729] T 594137cda8be4137beae728786a7731d P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:26:45.931231 2736 consensus_queue.cc:260] T 594137cda8be4137beae728786a7731d P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:45.933018 2736 ts_tablet_manager.cc:1398] T 594137cda8be4137beae728786a7731d P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 21:26:45.933861 2732 tablet_bootstrap.cc:492] T 9bc57b8a0388416290997df544e5cc0b P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:45.934072 2732 ts_tablet_manager.cc:1367] T 9bc57b8a0388416290997df544e5cc0b P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.107s user 0.004s sys 0.000s
I20240502 21:26:45.934886 2732 raft_consensus.cc:348] T 9bc57b8a0388416290997df544e5cc0b P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:45.935164 2732 raft_consensus.cc:374] T 9bc57b8a0388416290997df544e5cc0b P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:45.935287 2732 raft_consensus.cc:729] T 9bc57b8a0388416290997df544e5cc0b P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:45.935596 2732 consensus_queue.cc:260] T 9bc57b8a0388416290997df544e5cc0b P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:45.936362 2732 ts_tablet_manager.cc:1398] T 9bc57b8a0388416290997df544e5cc0b P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:45.939638 2724 tablet_copy_client.cc:308] T 182b3f15c06c4c07ade0a8854d3ecafd P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.130:37645
I20240502 21:26:45.944083 2738 ts_tablet_manager.cc:1888] T bb7211c63ddd450bb260a32dfb6110b5 P 1e04157b98dd4fad98858c290b73c3db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:45.945025 2738 log.cc:1196] T bb7211c63ddd450bb260a32dfb6110b5 P 1e04157b98dd4fad98858c290b73c3db: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal/wals/bb7211c63ddd450bb260a32dfb6110b5
I20240502 21:26:45.945513 1834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bc57b8a0388416290997df544e5cc0b" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
W20240502 21:26:45.946048 1901 leader_election.cc:341] T 9bc57b8a0388416290997df544e5cc0b P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:45.946247 1901 leader_election.cc:302] T 9bc57b8a0388416290997df544e5cc0b P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dec645b473f40578e726f3e50e219bf; no voters: 1ade0354c44c4c37ae480c7a94b11e1b, 1e04157b98dd4fad98858c290b73c3db
I20240502 21:26:45.946517 2515 raft_consensus.cc:2726] T 9bc57b8a0388416290997df544e5cc0b P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:26:45.944274 2732 tablet_bootstrap.cc:492] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:45.948207 2738 ts_tablet_manager.cc:1909] T bb7211c63ddd450bb260a32dfb6110b5 P 1e04157b98dd4fad98858c290b73c3db: Deleting consensus metadata
I20240502 21:26:45.952342 2729 tablet_bootstrap.cc:492] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:45.952575 2729 ts_tablet_manager.cc:1367] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.062s user 0.001s sys 0.004s
I20240502 21:26:45.953218 31381 catalog_manager.cc:4830] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): tablet bb7211c63ddd450bb260a32dfb6110b5 (table rebalancer_dynamic_table_new_name [id=85a71ef99c954778a60e20dc5a6b3d15]) successfully deleted
I20240502 21:26:45.953565 2729 raft_consensus.cc:348] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:45.953832 2729 raft_consensus.cc:374] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:45.953945 2729 raft_consensus.cc:729] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:45.954262 2729 consensus_queue.cc:260] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:45.954874 2729 ts_tablet_manager.cc:1398] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:45.955132 2729 tablet_bootstrap.cc:492] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:45.956568 2732 tablet_bootstrap.cc:654] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:45.957574 2729 tablet_bootstrap.cc:654] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:45.960821 2726 tablet_copy_client.cc:308] T bb64e0d78397429d959535f570658594 P 9dec645b473f40578e726f3e50e219bf: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.132:35933
I20240502 21:26:45.961349 2359 raft_consensus.cc:482] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:45.961483 2359 raft_consensus.cc:504] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:45.962059 2359 leader_election.cc:288] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:45.962620 2096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bc57b8a0388416290997df544e5cc0b" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:45.962782 1966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bc57b8a0388416290997df544e5cc0b" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:26:45.962834 2096 raft_consensus.cc:2454] T 9bc57b8a0388416290997df544e5cc0b P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 0.
I20240502 21:26:45.962985 1966 raft_consensus.cc:2454] T 9bc57b8a0388416290997df544e5cc0b P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 0.
I20240502 21:26:45.963433 1769 leader_election.cc:302] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 1e04157b98dd4fad98858c290b73c3db; no voters:
I20240502 21:26:45.963912 2359 raft_consensus.cc:2781] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:45.964033 2359 raft_consensus.cc:482] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:45.964159 2359 raft_consensus.cc:3037] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:45.966995 2359 raft_consensus.cc:504] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:45.967993 1966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bc57b8a0388416290997df544e5cc0b" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf"
I20240502 21:26:45.968187 1966 raft_consensus.cc:3037] T 9bc57b8a0388416290997df544e5cc0b P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:45.970458 31947 tablet_copy_service.cc:140] P 2d50f33557944bb480ea32199d9ee4f1: Received BeginTabletCopySession request for tablet bb64e0d78397429d959535f570658594 from peer 9dec645b473f40578e726f3e50e219bf ({username='slave'} at 127.24.226.134:56721)
W20240502 21:26:45.970647 31947 tablet_copy_service.cc:478] P 2d50f33557944bb480ea32199d9ee4f1: Error handling TabletCopyService RPC request from {username='slave'} at 127.24.226.134:56721: Unable to find specified tablet: bb64e0d78397429d959535f570658594: Not found: Tablet not found: bb64e0d78397429d959535f570658594
I20240502 21:26:45.970862 1966 raft_consensus.cc:2454] T 9bc57b8a0388416290997df544e5cc0b P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:26:45.971724 1767 leader_election.cc:302] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:26:45.972002 2763 raft_consensus.cc:2781] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:45.972621 2096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bc57b8a0388416290997df544e5cc0b" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:26:45.972858 2096 raft_consensus.cc:3037] T 9bc57b8a0388416290997df544e5cc0b P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:45.972973 2730 tablet_bootstrap.cc:492] T f02f83ec0a494da5b66241d9b379a03f P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:45.976454 2096 raft_consensus.cc:2454] T 9bc57b8a0388416290997df544e5cc0b P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:26:45.978976 2359 leader_election.cc:288] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Requested vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:45.985040 2747 tablet_copy_client.cc:308] T 49acc5301b9849e9b6795179acbc59a8 P 2d50f33557944bb480ea32199d9ee4f1: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.130:37645
I20240502 21:26:45.988965 2732 tablet_bootstrap.cc:492] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:45.989552 2732 ts_tablet_manager.cc:1367] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.045s user 0.000s sys 0.008s
I20240502 21:26:45.990276 2730 tablet_bootstrap.cc:492] T f02f83ec0a494da5b66241d9b379a03f P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:45.990725 2730 tablet_bootstrap.cc:492] T f02f83ec0a494da5b66241d9b379a03f P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap complete.
I20240502 21:26:45.990943 2730 ts_tablet_manager.cc:1367] T f02f83ec0a494da5b66241d9b379a03f P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.000s
I20240502 21:26:45.991427 2732 raft_consensus.cc:348] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:45.991624 2730 raft_consensus.cc:348] T f02f83ec0a494da5b66241d9b379a03f P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } }
I20240502 21:26:45.991838 2730 raft_consensus.cc:729] T f02f83ec0a494da5b66241d9b379a03f P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: LEARNER
I20240502 21:26:45.992048 2730 consensus_queue.cc:260] T f02f83ec0a494da5b66241d9b379a03f P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } }
I20240502 21:26:45.992693 2730 ts_tablet_manager.cc:1398] T f02f83ec0a494da5b66241d9b379a03f P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 21:26:45.993700 31687 tablet_copy_service.cc:140] P 05f4fac41c4f4959b9b316cb112d0ada: Received BeginTabletCopySession request for tablet 49acc5301b9849e9b6795179acbc59a8 from peer 2d50f33557944bb480ea32199d9ee4f1 ({username='slave'} at 127.24.226.132:43961)
I20240502 21:26:45.993927 31687 tablet_copy_service.cc:161] P 05f4fac41c4f4959b9b316cb112d0ada: Beginning new tablet copy session on tablet 49acc5301b9849e9b6795179acbc59a8 from peer 2d50f33557944bb480ea32199d9ee4f1 at {username='slave'} at 127.24.226.132:43961: session id = 2d50f33557944bb480ea32199d9ee4f1-49acc5301b9849e9b6795179acbc59a8
I20240502 21:26:45.995376 31686 tablet_copy_service.cc:140] P 05f4fac41c4f4959b9b316cb112d0ada: Received BeginTabletCopySession request for tablet 182b3f15c06c4c07ade0a8854d3ecafd from peer 1e04157b98dd4fad98858c290b73c3db ({username='slave'} at 127.24.226.133:58985)
I20240502 21:26:45.995568 31686 tablet_copy_service.cc:161] P 05f4fac41c4f4959b9b316cb112d0ada: Beginning new tablet copy session on tablet 182b3f15c06c4c07ade0a8854d3ecafd from peer 1e04157b98dd4fad98858c290b73c3db at {username='slave'} at 127.24.226.133:58985: session id = 1e04157b98dd4fad98858c290b73c3db-182b3f15c06c4c07ade0a8854d3ecafd
I20240502 21:26:45.995795 1884 raft_consensus.cc:482] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:45.995939 1884 raft_consensus.cc:504] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:45.996613 1884 leader_election.cc:288] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:26:45.997717 31687 tablet_copy_source_session.cc:205] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:45.997692 1966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "594137cda8be4137beae728786a7731d" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:26:45.997881 1966 raft_consensus.cc:2454] T 594137cda8be4137beae728786a7731d P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:45.998255 31686 tablet_copy_source_session.cc:205] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:45.998287 31534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "594137cda8be4137beae728786a7731d" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:45.998483 31534 raft_consensus.cc:2454] T 594137cda8be4137beae728786a7731d P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:45.999706 2729 tablet_bootstrap.cc:492] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:45.999831 2729 ts_tablet_manager.cc:1367] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.045s user 0.000s sys 0.007s
I20240502 21:26:46.000365 31863 leader_election.cc:302] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:46.000507 2732 raft_consensus.cc:374] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:46.000578 2732 raft_consensus.cc:729] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:46.000623 2724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182b3f15c06c4c07ade0a8854d3ecafd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:46.000792 2732 consensus_queue.cc:260] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:46.001333 2732 ts_tablet_manager.cc:1398] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.011s user 0.000s sys 0.001s
I20240502 21:26:46.001444 2763 raft_consensus.cc:686] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Becoming Leader. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Running, Role: LEADER
I20240502 21:26:46.001739 2763 consensus_queue.cc:237] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:46.002475 2747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49acc5301b9849e9b6795179acbc59a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:46.002833 2729 raft_consensus.cc:348] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:46.002991 2729 raft_consensus.cc:374] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:46.003062 2729 raft_consensus.cc:729] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:46.003278 2729 consensus_queue.cc:260] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:46.003826 2729 ts_tablet_manager.cc:1398] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:26:46.006961 2640 raft_consensus.cc:2781] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:46.007054 2640 raft_consensus.cc:482] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:46.007184 2640 raft_consensus.cc:3037] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:46.007200 2724 tablet_copy_client.cc:783] T 182b3f15c06c4c07ade0a8854d3ecafd P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:46.007537 2724 tablet_copy_client.cc:647] T 182b3f15c06c4c07ade0a8854d3ecafd P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:46.009805 2724 tablet_copy_client.cc:523] T 182b3f15c06c4c07ade0a8854d3ecafd P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:46.009840 2640 raft_consensus.cc:504] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:46.011142 1986 tablet_copy_service.cc:342] P 9dec645b473f40578e726f3e50e219bf: Request end of tablet copy session a44f41b0b6644fb6aa29476aed8c964f-f02f83ec0a494da5b66241d9b379a03f received from {username='slave'} at 127.24.226.129:41327
I20240502 21:26:46.011317 1986 tablet_copy_service.cc:433] P 9dec645b473f40578e726f3e50e219bf: ending tablet copy session a44f41b0b6644fb6aa29476aed8c964f-f02f83ec0a494da5b66241d9b379a03f on tablet f02f83ec0a494da5b66241d9b379a03f with peer a44f41b0b6644fb6aa29476aed8c964f
I20240502 21:26:46.021018 2724 tablet_bootstrap.cc:492] T 182b3f15c06c4c07ade0a8854d3ecafd P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:46.031481 1966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "594137cda8be4137beae728786a7731d" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf"
I20240502 21:26:46.031682 1966 raft_consensus.cc:3037] T 594137cda8be4137beae728786a7731d P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:46.034421 1966 raft_consensus.cc:2454] T 594137cda8be4137beae728786a7731d P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:26:46.035727 31860 leader_election.cc:302] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:26:46.036818 1884 raft_consensus.cc:2781] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:46.036979 1884 raft_consensus.cc:686] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Running, Role: LEADER
I20240502 21:26:46.037292 1884 consensus_queue.cc:237] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:46.039093 2640 leader_election.cc:288] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Requested vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:26:46.039157 31534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "594137cda8be4137beae728786a7731d" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:26:46.039386 31534 raft_consensus.cc:3037] T 594137cda8be4137beae728786a7731d P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:46.040407 2747 tablet_copy_client.cc:783] T 49acc5301b9849e9b6795179acbc59a8 P 2d50f33557944bb480ea32199d9ee4f1: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:46.040740 2747 tablet_copy_client.cc:647] T 49acc5301b9849e9b6795179acbc59a8 P 2d50f33557944bb480ea32199d9ee4f1: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:46.043722 31534 raft_consensus.cc:2454] T 594137cda8be4137beae728786a7731d P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:26:46.044601 2747 tablet_copy_client.cc:523] T 49acc5301b9849e9b6795179acbc59a8 P 2d50f33557944bb480ea32199d9ee4f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:46.052469 31393 catalog_manager.cc:5484] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132). New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:46.055634 2747 tablet_bootstrap.cc:492] T 49acc5301b9849e9b6795179acbc59a8 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:46.067734 2303 raft_consensus.cc:482] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:46.067878 2303 raft_consensus.cc:504] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:46.067924 31393 catalog_manager.cc:5484] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e04157b98dd4fad98858c290b73c3db (127.24.226.133). New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:46.068763 2303 leader_election.cc:288] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:46.068997 1834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc6c7211d5f4f73b2a050b9ec78c6b1" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:26:46.069250 1834 raft_consensus.cc:2454] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:46.069588 2096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc6c7211d5f4f73b2a050b9ec78c6b1" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:46.069882 2096 raft_consensus.cc:2454] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:46.069969 2732 tablet_bootstrap.cc:492] T 577e7a586ad9459f8e77ce4b8f563c09 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:46.070313 31471 leader_election.cc:302] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:46.070885 2303 raft_consensus.cc:2781] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:46.070976 2303 raft_consensus.cc:482] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:46.071067 2303 raft_consensus.cc:3037] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:46.072749 2732 tablet_bootstrap.cc:654] T 577e7a586ad9459f8e77ce4b8f563c09 P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:46.073688 2303 raft_consensus.cc:504] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:46.074729 1834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc6c7211d5f4f73b2a050b9ec78c6b1" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:26:46.074930 1834 raft_consensus.cc:3037] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:46.075084 2303 leader_election.cc:288] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Requested vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:46.074930 2096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc6c7211d5f4f73b2a050b9ec78c6b1" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:26:46.077466 2096 raft_consensus.cc:3037] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:46.078157 1834 raft_consensus.cc:2454] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:46.078609 31471 leader_election.cc:302] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:46.079077 2303 raft_consensus.cc:2781] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:46.079272 2303 raft_consensus.cc:686] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:26:46.079571 2303 consensus_queue.cc:237] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:46.079880 2096 raft_consensus.cc:2454] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:46.102627 31393 catalog_manager.cc:5484] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 0 to 1, leader changed from <none> to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:46.105259 2732 tablet_bootstrap.cc:492] T 577e7a586ad9459f8e77ce4b8f563c09 P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:46.105414 2732 ts_tablet_manager.cc:1367] T 577e7a586ad9459f8e77ce4b8f563c09 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.036s user 0.001s sys 0.004s
I20240502 21:26:46.106189 2732 raft_consensus.cc:348] T 577e7a586ad9459f8e77ce4b8f563c09 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:46.106366 2732 raft_consensus.cc:374] T 577e7a586ad9459f8e77ce4b8f563c09 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:46.106452 2732 raft_consensus.cc:729] T 577e7a586ad9459f8e77ce4b8f563c09 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:46.106680 2732 consensus_queue.cc:260] T 577e7a586ad9459f8e77ce4b8f563c09 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:46.107839 2732 ts_tablet_manager.cc:1398] T 577e7a586ad9459f8e77ce4b8f563c09 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:46.110420 2747 tablet_bootstrap.cc:492] T 49acc5301b9849e9b6795179acbc59a8 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:46.111070 2747 tablet_bootstrap.cc:492] T 49acc5301b9849e9b6795179acbc59a8 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap complete.
I20240502 21:26:46.111440 2747 ts_tablet_manager.cc:1367] T 49acc5301b9849e9b6795179acbc59a8 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.056s user 0.019s sys 0.007s
I20240502 21:26:46.112385 2747 raft_consensus.cc:348] T 49acc5301b9849e9b6795179acbc59a8 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } }
I20240502 21:26:46.112705 2747 raft_consensus.cc:729] T 49acc5301b9849e9b6795179acbc59a8 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: LEARNER
I20240502 21:26:46.112958 2747 consensus_queue.cc:260] T 49acc5301b9849e9b6795179acbc59a8 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } }
I20240502 21:26:46.114961 2747 ts_tablet_manager.cc:1398] T 49acc5301b9849e9b6795179acbc59a8 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:26:46.115708 31686 tablet_copy_service.cc:342] P 05f4fac41c4f4959b9b316cb112d0ada: Request end of tablet copy session 2d50f33557944bb480ea32199d9ee4f1-49acc5301b9849e9b6795179acbc59a8 received from {username='slave'} at 127.24.226.132:43961
I20240502 21:26:46.115898 31686 tablet_copy_service.cc:433] P 05f4fac41c4f4959b9b316cb112d0ada: ending tablet copy session 2d50f33557944bb480ea32199d9ee4f1-49acc5301b9849e9b6795179acbc59a8 on tablet 49acc5301b9849e9b6795179acbc59a8 with peer 2d50f33557944bb480ea32199d9ee4f1
I20240502 21:26:46.185807 31795 consensus_queue.cc:237] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:26:46.187747 2724 tablet_bootstrap.cc:492] T 182b3f15c06c4c07ade0a8854d3ecafd P 1e04157b98dd4fad98858c290b73c3db: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:46.188640 2724 tablet_bootstrap.cc:492] T 182b3f15c06c4c07ade0a8854d3ecafd P 1e04157b98dd4fad98858c290b73c3db: Bootstrap complete.
I20240502 21:26:46.188974 2724 ts_tablet_manager.cc:1367] T 182b3f15c06c4c07ade0a8854d3ecafd P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.168s user 0.046s sys 0.011s
I20240502 21:26:46.189842 2724 raft_consensus.cc:348] T 182b3f15c06c4c07ade0a8854d3ecafd P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } }
I20240502 21:26:46.190140 2724 raft_consensus.cc:729] T 182b3f15c06c4c07ade0a8854d3ecafd P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: LEARNER
I20240502 21:26:46.190387 2724 consensus_queue.cc:260] T 182b3f15c06c4c07ade0a8854d3ecafd P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } }
I20240502 21:26:46.190794 31535 raft_consensus.cc:1203] T f02f83ec0a494da5b66241d9b379a03f P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:26:46.191571 2724 ts_tablet_manager.cc:1398] T 182b3f15c06c4c07ade0a8854d3ecafd P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 21:26:46.192924 31686 tablet_copy_service.cc:342] P 05f4fac41c4f4959b9b316cb112d0ada: Request end of tablet copy session 1e04157b98dd4fad98858c290b73c3db-182b3f15c06c4c07ade0a8854d3ecafd received from {username='slave'} at 127.24.226.133:58985
I20240502 21:26:46.193078 31686 tablet_copy_service.cc:433] P 05f4fac41c4f4959b9b316cb112d0ada: ending tablet copy session 1e04157b98dd4fad98858c290b73c3db-182b3f15c06c4c07ade0a8854d3ecafd on tablet 182b3f15c06c4c07ade0a8854d3ecafd with peer 1e04157b98dd4fad98858c290b73c3db
I20240502 21:26:46.198160 31389 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:26:46.199121 31926 raft_consensus.cc:1261] T b407c2c9b3a14ce3ada52b7856c6ecee P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:46.199368 31666 raft_consensus.cc:1261] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:46.199939 2791 consensus_queue.cc:1035] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:46.200384 2416 consensus_queue.cc:1035] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:46.208738 31534 raft_consensus.cc:1261] T 594137cda8be4137beae728786a7731d P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:46.208951 1966 raft_consensus.cc:1261] T 594137cda8be4137beae728786a7731d P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:46.209457 2640 consensus_queue.cc:1035] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:46.209750 1884 consensus_queue.cc:1035] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:46.214720 31730 consensus_peers.cc:485] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b407c2c9b3a14ce3ada52b7856c6ecee. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:46.214226 2799 raft_consensus.cc:2932] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:46.215018 31389 catalog_manager.cc:5576] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753) does not have the latest schema for tablet 594137cda8be4137beae728786a7731d (table rebalancer_dynamic_table [id=8123e6cbc25b44e1899a876d634dfaa5]). Expected version 1 got 0
I20240502 21:26:46.217973 31926 raft_consensus.cc:2932] T b407c2c9b3a14ce3ada52b7856c6ecee P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:46.226799 31389 catalog_manager.cc:5484] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:46.227344 31666 raft_consensus.cc:2932] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:46.235574 31390 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240502 21:26:46.235801 31390 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240502 21:26:46.240612 31390 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 1e9b7eae127146a18a27d26d103b04d3
I20240502 21:26:46.241127 31390 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet e69d22e2c0b348b1bc439c1618bd59f6
I20240502 21:26:46.241377 2801 tablet.cc:1722] T 594137cda8be4137beae728786a7731d P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:46.241539 31390 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet d3b744acac964cbd82abc3306f016baf
I20240502 21:26:46.241838 31775 tablet_service.cc:1514] Processing DeleteTablet for tablet e69d22e2c0b348b1bc439c1618bd59f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:46 UTC) from {username='slave'} at 127.0.0.1:51596
I20240502 21:26:46.242087 2739 tablet_replica.cc:330] T e69d22e2c0b348b1bc439c1618bd59f6 P 08d7a0680d244b55b93d17f9ab57b84c: stopping tablet replica
I20240502 21:26:46.242280 2739 raft_consensus.cc:2229] T e69d22e2c0b348b1bc439c1618bd59f6 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:46.242585 2739 raft_consensus.cc:2258] T e69d22e2c0b348b1bc439c1618bd59f6 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:46.242964 2800 tablet.cc:1722] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:46.243374 2739 ts_tablet_manager.cc:1875] T e69d22e2c0b348b1bc439c1618bd59f6 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:46.243567 2800 tablet.cc:1620] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1: MemRowSet was empty: no flush needed.
I20240502 21:26:46.244949 2801 tablet.cc:1620] T 594137cda8be4137beae728786a7731d P 9dec645b473f40578e726f3e50e219bf: MemRowSet was empty: no flush needed.
I20240502 21:26:46.246872 31645 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e9b7eae127146a18a27d26d103b04d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:46 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:46.247032 31515 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e9b7eae127146a18a27d26d103b04d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:46 UTC) from {username='slave'} at 127.0.0.1:45194
I20240502 21:26:46.246868 31647 tablet_service.cc:1514] Processing DeleteTablet for tablet e69d22e2c0b348b1bc439c1618bd59f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:46 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:46.247265 2602 tablet_replica.cc:330] T 1e9b7eae127146a18a27d26d103b04d3 P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:46.247488 2602 raft_consensus.cc:2229] T 1e9b7eae127146a18a27d26d103b04d3 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:46.247658 31907 tablet_service.cc:1514] Processing DeleteTablet for tablet e69d22e2c0b348b1bc439c1618bd59f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:46 UTC) from {username='slave'} at 127.0.0.1:47824
I20240502 21:26:46.247749 2602 raft_consensus.cc:2258] T 1e9b7eae127146a18a27d26d103b04d3 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:46.247790 31906 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e9b7eae127146a18a27d26d103b04d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:46 UTC) from {username='slave'} at 127.0.0.1:47824
I20240502 21:26:46.247982 31905 tablet_service.cc:1514] Processing DeleteTablet for tablet d3b744acac964cbd82abc3306f016baf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:46 UTC) from {username='slave'} at 127.0.0.1:47824
I20240502 21:26:46.248703 2328 tablet_replica.cc:330] T e69d22e2c0b348b1bc439c1618bd59f6 P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:26:46.248798 31394 catalog_manager.cc:5576] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753) does not have the latest schema for tablet 594137cda8be4137beae728786a7731d (table rebalancer_dynamic_table [id=8123e6cbc25b44e1899a876d634dfaa5]). Expected version 1 got 0
I20240502 21:26:46.248940 2328 raft_consensus.cc:2229] T e69d22e2c0b348b1bc439c1618bd59f6 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:46.249200 2328 raft_consensus.cc:2258] T e69d22e2c0b348b1bc439c1618bd59f6 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:46.249879 2328 ts_tablet_manager.cc:1875] T e69d22e2c0b348b1bc439c1618bd59f6 P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:46.252210 2329 tablet_replica.cc:330] T 1e9b7eae127146a18a27d26d103b04d3 P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:46.252524 2329 raft_consensus.cc:2229] T 1e9b7eae127146a18a27d26d103b04d3 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:46.252732 31515 tablet_service.cc:1514] Processing DeleteTablet for tablet d3b744acac964cbd82abc3306f016baf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:46 UTC) from {username='slave'} at 127.0.0.1:45194
I20240502 21:26:46.252887 2329 raft_consensus.cc:2258] T 1e9b7eae127146a18a27d26d103b04d3 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:46.253170 31647 tablet_service.cc:1514] Processing DeleteTablet for tablet d3b744acac964cbd82abc3306f016baf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:46 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:46.253645 2329 ts_tablet_manager.cc:1875] T 1e9b7eae127146a18a27d26d103b04d3 P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:46.259102 2602 ts_tablet_manager.cc:1875] T 1e9b7eae127146a18a27d26d103b04d3 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:46.261875 2739 ts_tablet_manager.cc:1888] T e69d22e2c0b348b1bc439c1618bd59f6 P 08d7a0680d244b55b93d17f9ab57b84c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:46.262053 2739 log.cc:1196] T e69d22e2c0b348b1bc439c1618bd59f6 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/wals/e69d22e2c0b348b1bc439c1618bd59f6
I20240502 21:26:46.262614 2739 ts_tablet_manager.cc:1909] T e69d22e2c0b348b1bc439c1618bd59f6 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting consensus metadata
I20240502 21:26:46.263747 31381 catalog_manager.cc:4830] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): tablet e69d22e2c0b348b1bc439c1618bd59f6 (table rebalance_test_table_14 [id=c880d77c61494966b13524216c035e27]) successfully deleted
I20240502 21:26:46.267652 2602 ts_tablet_manager.cc:1888] T 1e9b7eae127146a18a27d26d103b04d3 P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:46.267796 2602 log.cc:1196] T 1e9b7eae127146a18a27d26d103b04d3 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/1e9b7eae127146a18a27d26d103b04d3
I20240502 21:26:46.268235 2602 ts_tablet_manager.cc:1909] T 1e9b7eae127146a18a27d26d103b04d3 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:26:46.272145 2329 ts_tablet_manager.cc:1888] T 1e9b7eae127146a18a27d26d103b04d3 P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:46.272325 2329 log.cc:1196] T 1e9b7eae127146a18a27d26d103b04d3 P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/1e9b7eae127146a18a27d26d103b04d3
I20240502 21:26:46.273582 2328 ts_tablet_manager.cc:1888] T e69d22e2c0b348b1bc439c1618bd59f6 P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:46.273710 2328 log.cc:1196] T e69d22e2c0b348b1bc439c1618bd59f6 P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/e69d22e2c0b348b1bc439c1618bd59f6
I20240502 21:26:46.274149 2328 ts_tablet_manager.cc:1909] T e69d22e2c0b348b1bc439c1618bd59f6 P 2d50f33557944bb480ea32199d9ee4f1: Deleting consensus metadata
I20240502 21:26:46.275147 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet e69d22e2c0b348b1bc439c1618bd59f6 (table rebalance_test_table_14 [id=c880d77c61494966b13524216c035e27]) successfully deleted
I20240502 21:26:46.275454 2328 tablet_replica.cc:330] T 1e9b7eae127146a18a27d26d103b04d3 P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:26:46.275689 2328 raft_consensus.cc:2229] T 1e9b7eae127146a18a27d26d103b04d3 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:46.275882 2328 raft_consensus.cc:2258] T 1e9b7eae127146a18a27d26d103b04d3 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:46.276512 2328 ts_tablet_manager.cc:1875] T 1e9b7eae127146a18a27d26d103b04d3 P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:46.281354 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet 1e9b7eae127146a18a27d26d103b04d3 (table rebalance_test_table_14 [id=c880d77c61494966b13524216c035e27]) successfully deleted
I20240502 21:26:46.281697 2329 ts_tablet_manager.cc:1909] T 1e9b7eae127146a18a27d26d103b04d3 P a44f41b0b6644fb6aa29476aed8c964f: Deleting consensus metadata
I20240502 21:26:46.281949 2602 tablet_replica.cc:330] T e69d22e2c0b348b1bc439c1618bd59f6 P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:46.282210 2602 raft_consensus.cc:2229] T e69d22e2c0b348b1bc439c1618bd59f6 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:46.283293 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet 1e9b7eae127146a18a27d26d103b04d3 (table rebalance_test_table_14 [id=c880d77c61494966b13524216c035e27]) successfully deleted
I20240502 21:26:46.283716 2602 raft_consensus.cc:2258] T e69d22e2c0b348b1bc439c1618bd59f6 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:46.284756 2602 ts_tablet_manager.cc:1875] T e69d22e2c0b348b1bc439c1618bd59f6 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:46.285030 2329 tablet_replica.cc:330] T d3b744acac964cbd82abc3306f016baf P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:46.285284 2329 raft_consensus.cc:2229] T d3b744acac964cbd82abc3306f016baf P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:46.285542 2329 raft_consensus.cc:2258] T d3b744acac964cbd82abc3306f016baf P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:46.285805 31535 consensus_queue.cc:237] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } }
I20240502 21:26:46.286998 2329 ts_tablet_manager.cc:1875] T d3b744acac964cbd82abc3306f016baf P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:46.293263 31394 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=8123e6cbc25b44e1899a876d634dfaa5] alter complete (version 1)
I20240502 21:26:46.300365 2328 ts_tablet_manager.cc:1888] T 1e9b7eae127146a18a27d26d103b04d3 P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
W20240502 21:26:46.300709 31471 consensus_peers.cc:485] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f -> Peer 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): Couldn't send request to peer 1ade0354c44c4c37ae480c7a94b11e1b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe4e41225da24d7c91c6bf7733a5e067. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:46.300904 31666 raft_consensus.cc:1261] T fe4e41225da24d7c91c6bf7733a5e067 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:46.300400 31926 raft_consensus.cc:1261] T fe4e41225da24d7c91c6bf7733a5e067 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:46.301527 2631 consensus_queue.cc:1035] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:46.302035 2303 consensus_queue.cc:1035] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:46.303997 2328 log.cc:1196] T 1e9b7eae127146a18a27d26d103b04d3 P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/1e9b7eae127146a18a27d26d103b04d3
I20240502 21:26:46.304508 2328 ts_tablet_manager.cc:1909] T 1e9b7eae127146a18a27d26d103b04d3 P 2d50f33557944bb480ea32199d9ee4f1: Deleting consensus metadata
I20240502 21:26:46.305375 2328 tablet_replica.cc:330] T d3b744acac964cbd82abc3306f016baf P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:26:46.305545 2328 raft_consensus.cc:2229] T d3b744acac964cbd82abc3306f016baf P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:46.305220 2303 raft_consensus.cc:2932] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } } }
I20240502 21:26:46.305631 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet 1e9b7eae127146a18a27d26d103b04d3 (table rebalance_test_table_14 [id=c880d77c61494966b13524216c035e27]) successfully deleted
I20240502 21:26:46.305821 2328 raft_consensus.cc:2258] T d3b744acac964cbd82abc3306f016baf P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:46.306396 2328 ts_tablet_manager.cc:1875] T d3b744acac964cbd82abc3306f016baf P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:46.306069 31666 raft_consensus.cc:2932] T fe4e41225da24d7c91c6bf7733a5e067 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } } }
I20240502 21:26:46.306442 31390 catalog_manager.cc:5576] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039) does not have the latest schema for tablet 594137cda8be4137beae728786a7731d (table rebalancer_dynamic_table [id=8123e6cbc25b44e1899a876d634dfaa5]). Expected version 1 got 0
I20240502 21:26:46.311862 2328 ts_tablet_manager.cc:1888] T d3b744acac964cbd82abc3306f016baf P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:46.311980 2328 log.cc:1196] T d3b744acac964cbd82abc3306f016baf P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/d3b744acac964cbd82abc3306f016baf
I20240502 21:26:46.312577 2328 ts_tablet_manager.cc:1909] T d3b744acac964cbd82abc3306f016baf P 2d50f33557944bb480ea32199d9ee4f1: Deleting consensus metadata
I20240502 21:26:46.313519 31394 catalog_manager.cc:5484] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:46.315480 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet d3b744acac964cbd82abc3306f016baf (table rebalance_test_table_14 [id=c880d77c61494966b13524216c035e27]) successfully deleted
I20240502 21:26:46.315758 2329 ts_tablet_manager.cc:1888] T d3b744acac964cbd82abc3306f016baf P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:46.315862 2329 log.cc:1196] T d3b744acac964cbd82abc3306f016baf P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/d3b744acac964cbd82abc3306f016baf
I20240502 21:26:46.316298 2329 ts_tablet_manager.cc:1909] T d3b744acac964cbd82abc3306f016baf P a44f41b0b6644fb6aa29476aed8c964f: Deleting consensus metadata
I20240502 21:26:46.318634 2602 ts_tablet_manager.cc:1888] T e69d22e2c0b348b1bc439c1618bd59f6 P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:46.318753 2602 log.cc:1196] T e69d22e2c0b348b1bc439c1618bd59f6 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/e69d22e2c0b348b1bc439c1618bd59f6
I20240502 21:26:46.319149 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet d3b744acac964cbd82abc3306f016baf (table rebalance_test_table_14 [id=c880d77c61494966b13524216c035e27]) successfully deleted
I20240502 21:26:46.319485 2602 ts_tablet_manager.cc:1909] T e69d22e2c0b348b1bc439c1618bd59f6 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:26:46.320600 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet e69d22e2c0b348b1bc439c1618bd59f6 (table rebalance_test_table_14 [id=c880d77c61494966b13524216c035e27]) successfully deleted
I20240502 21:26:46.321249 2602 tablet_replica.cc:330] T d3b744acac964cbd82abc3306f016baf P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:46.321501 2602 raft_consensus.cc:2229] T d3b744acac964cbd82abc3306f016baf P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:46.321738 2602 raft_consensus.cc:2258] T d3b744acac964cbd82abc3306f016baf P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:46.324429 2602 ts_tablet_manager.cc:1875] T d3b744acac964cbd82abc3306f016baf P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:46.326406 31924 raft_consensus.cc:2932] T fe4e41225da24d7c91c6bf7733a5e067 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } } }
I20240502 21:26:46.330701 2813 tablet.cc:1722] T 594137cda8be4137beae728786a7731d P a44f41b0b6644fb6aa29476aed8c964f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:46.331167 2602 ts_tablet_manager.cc:1888] T d3b744acac964cbd82abc3306f016baf P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:46.331315 2602 log.cc:1196] T d3b744acac964cbd82abc3306f016baf P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/d3b744acac964cbd82abc3306f016baf
I20240502 21:26:46.331698 2602 ts_tablet_manager.cc:1909] T d3b744acac964cbd82abc3306f016baf P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:26:46.332659 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet d3b744acac964cbd82abc3306f016baf (table rebalance_test_table_14 [id=c880d77c61494966b13524216c035e27]) successfully deleted
I20240502 21:26:46.333591 31390 catalog_manager.cc:5576] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039) does not have the latest schema for tablet 594137cda8be4137beae728786a7731d (table rebalancer_dynamic_table [id=8123e6cbc25b44e1899a876d634dfaa5]). Expected version 1 got 0
I20240502 21:26:46.335568 2813 tablet.cc:1620] T 594137cda8be4137beae728786a7731d P a44f41b0b6644fb6aa29476aed8c964f: MemRowSet was empty: no flush needed.
I20240502 21:26:46.342612 2763 consensus_queue.cc:1035] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:46.349475 2096 raft_consensus.cc:3037] T 577e7a586ad9459f8e77ce4b8f563c09 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:46.353228 2359 consensus_queue.cc:1035] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:46.377489 31926 consensus_queue.cc:237] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } }
I20240502 21:26:46.380213 31534 raft_consensus.cc:1261] T 62f083e9641042c990efcfb1c4104de5 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:46.382694 2777 consensus_queue.cc:1035] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:46.395131 31390 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:26:46.395867 31795 raft_consensus.cc:1261] T 62f083e9641042c990efcfb1c4104de5 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:46.398157 2777 consensus_queue.cc:1035] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:46.400933 1884 raft_consensus.cc:2932] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } } }
I20240502 21:26:46.402025 31534 raft_consensus.cc:2932] T 62f083e9641042c990efcfb1c4104de5 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } } }
I20240502 21:26:46.402413 31795 raft_consensus.cc:2932] T 62f083e9641042c990efcfb1c4104de5 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } } }
I20240502 21:26:46.409032 31390 catalog_manager.cc:5484] T 62f083e9641042c990efcfb1c4104de5 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: config changed from index -1 to 4, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } } }
I20240502 21:26:46.409405 2801 tablet.cc:1722] T 594137cda8be4137beae728786a7731d P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:46.409837 2801 tablet.cc:1620] T 594137cda8be4137beae728786a7731d P 9dec645b473f40578e726f3e50e219bf: MemRowSet was empty: no flush needed.
I20240502 21:26:46.410096 2800 tablet.cc:1722] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:46.410557 2800 tablet.cc:1620] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1: MemRowSet was empty: no flush needed.
I20240502 21:26:46.413731 2813 tablet.cc:1722] T 594137cda8be4137beae728786a7731d P a44f41b0b6644fb6aa29476aed8c964f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:46.414263 2813 tablet.cc:1620] T 594137cda8be4137beae728786a7731d P a44f41b0b6644fb6aa29476aed8c964f: MemRowSet was empty: no flush needed.
W20240502 21:26:46.415791 31862 consensus_peers.cc:485] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 -> Peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): Couldn't send request to peer d1406e92aec84ac585304e57bd37a40d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62f083e9641042c990efcfb1c4104de5. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:46.418956 31926 raft_consensus.cc:1203] T 49acc5301b9849e9b6795179acbc59a8 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:26:46.423080 31394 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=8123e6cbc25b44e1899a876d634dfaa5] alter complete (version 2)
I20240502 21:26:46.436371 31795 consensus_queue.cc:237] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } }
I20240502 21:26:46.438539 31535 raft_consensus.cc:1261] T 815b1fc784714c8cb4ba76707c345625 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:46.438722 1834 raft_consensus.cc:1261] T 815b1fc784714c8cb4ba76707c345625 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:46.438987 2416 consensus_queue.cc:1035] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:46.439324 2791 consensus_queue.cc:1035] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:26:46.439581 31733 consensus_peers.cc:485] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c -> Peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): Couldn't send request to peer 2d50f33557944bb480ea32199d9ee4f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 815b1fc784714c8cb4ba76707c345625. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:46.441408 2416 raft_consensus.cc:2932] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } } }
I20240502 21:26:46.441943 31535 raft_consensus.cc:2932] T 815b1fc784714c8cb4ba76707c345625 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } } }
I20240502 21:26:46.442543 1834 raft_consensus.cc:2932] T 815b1fc784714c8cb4ba76707c345625 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } } }
I20240502 21:26:46.447059 31394 catalog_manager.cc:5484] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index -1 to 2, NON_VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) added. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:46.461366 31535 consensus_queue.cc:237] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:26:46.465607 31795 raft_consensus.cc:1261] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:46.465705 31666 raft_consensus.cc:1261] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:26:46.465898 31469 consensus_peers.cc:485] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 55fc18e4cc98446ca50c1dd2b6a9a89d. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:46.466190 2809 consensus_queue.cc:1035] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:46.466408 2303 consensus_queue.cc:1035] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:46.469048 2303 raft_consensus.cc:2932] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:46.469588 31795 raft_consensus.cc:2932] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:46.471050 31666 raft_consensus.cc:2932] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:46.474783 2631 consensus_queue.cc:1035] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:46.474975 31393 catalog_manager.cc:5484] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:46.491999 2773 consensus_queue.cc:1035] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:46.494397 2843 consensus_queue.cc:1035] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:46.507822 2359 consensus_queue.cc:1035] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:46.517680 1833 raft_consensus.cc:1203] T 182b3f15c06c4c07ade0a8854d3ecafd P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:26:46.526940 1833 consensus_queue.cc:237] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } }
I20240502 21:26:46.527895 31389 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:26:46.529873 2096 raft_consensus.cc:1261] T b9342dbe0497499ba4c238f2ed033293 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:46.530683 2359 consensus_queue.cc:1035] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:46.530958 31795 raft_consensus.cc:1261] T b9342dbe0497499ba4c238f2ed033293 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:26:46.531538 1770 consensus_peers.cc:485] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db -> Peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): Couldn't send request to peer 05f4fac41c4f4959b9b316cb112d0ada. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b9342dbe0497499ba4c238f2ed033293. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:46.532469 2359 consensus_queue.cc:1035] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:46.535123 2816 raft_consensus.cc:2932] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } } }
I20240502 21:26:46.537546 2800 tablet.cc:1722] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:46.538390 2813 tablet.cc:1722] T 594137cda8be4137beae728786a7731d P a44f41b0b6644fb6aa29476aed8c964f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:46.541435 31393 catalog_manager.cc:5484] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db reported cstate change: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:46.542505 2801 tablet.cc:1722] T 594137cda8be4137beae728786a7731d P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:46.550673 2096 raft_consensus.cc:2932] T b9342dbe0497499ba4c238f2ed033293 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } } }
I20240502 21:26:46.554653 31394 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table_new_name [id=8123e6cbc25b44e1899a876d634dfaa5] alter complete (version 3)
I20240502 21:26:46.555444 31795 raft_consensus.cc:2932] T b9342dbe0497499ba4c238f2ed033293 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } } }
I20240502 21:26:46.582146 31795 consensus_queue.cc:237] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } }
I20240502 21:26:46.584666 31535 raft_consensus.cc:1261] T 5b1dcae186f542cea92c9191fa8ef951 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:46.584733 31926 raft_consensus.cc:1261] T 5b1dcae186f542cea92c9191fa8ef951 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:46.585283 2416 consensus_queue.cc:1035] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:46.585865 2416 consensus_queue.cc:1035] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:46.587759 2416 raft_consensus.cc:2932] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } } }
I20240502 21:26:46.591010 31926 raft_consensus.cc:2932] T 5b1dcae186f542cea92c9191fa8ef951 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } } }
W20240502 21:26:46.594175 31732 consensus_peers.cc:485] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c -> Peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): Couldn't send request to peer d1406e92aec84ac585304e57bd37a40d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5b1dcae186f542cea92c9191fa8ef951. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:46.595837 31394 catalog_manager.cc:5484] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index -1 to 4, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:46.603478 31535 raft_consensus.cc:2932] T 5b1dcae186f542cea92c9191fa8ef951 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } } }
I20240502 21:26:46.621698 31795 consensus_queue.cc:237] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } }
W20240502 21:26:46.624107 31732 consensus_peers.cc:485] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c -> Peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): Couldn't send request to peer 1e04157b98dd4fad98858c290b73c3db. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9ea8ddb9f8e04893b56c10dbbb5af321. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:46.624375 31926 raft_consensus.cc:1261] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:46.624640 31666 raft_consensus.cc:1261] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:46.625028 2799 consensus_queue.cc:1035] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:46.625324 2791 consensus_queue.cc:1035] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:46.629827 2416 raft_consensus.cc:2932] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } } }
I20240502 21:26:46.630928 31666 raft_consensus.cc:2932] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } } }
I20240502 21:26:46.636070 31926 raft_consensus.cc:2932] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } } }
I20240502 21:26:46.636883 31389 catalog_manager.cc:5484] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index -1 to 4, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:46.644973 2872 ts_tablet_manager.cc:902] T b407c2c9b3a14ce3ada52b7856c6ecee P 9dec645b473f40578e726f3e50e219bf: Initiating tablet copy from peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:46.662496 2792 raft_consensus.cc:1050] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf: attempting to promote NON_VOTER a44f41b0b6644fb6aa29476aed8c964f to VOTER
I20240502 21:26:46.663273 31393 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:46.663549 31393 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:46.663437 2792 consensus_queue.cc:237] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } }
I20240502 21:26:46.666105 2096 raft_consensus.cc:1261] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:46.666801 2515 consensus_queue.cc:1035] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:46.667645 31535 raft_consensus.cc:1261] T f02f83ec0a494da5b66241d9b379a03f P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEARNER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:46.668071 31393 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 594137cda8be4137beae728786a7731d
I20240502 21:26:46.668125 2872 tablet_copy_client.cc:308] T b407c2c9b3a14ce3ada52b7856c6ecee P 9dec645b473f40578e726f3e50e219bf: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.131:40807
I20240502 21:26:46.668562 2515 consensus_queue.cc:1035] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:46.668754 31905 tablet_service.cc:1514] Processing DeleteTablet for tablet 594137cda8be4137beae728786a7731d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:46 UTC) from {username='slave'} at 127.0.0.1:47824
I20240502 21:26:46.669031 2328 tablet_replica.cc:330] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:26:46.669273 2328 raft_consensus.cc:2229] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:46.669600 2328 raft_consensus.cc:2258] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:46.670555 31515 tablet_service.cc:1514] Processing DeleteTablet for tablet 594137cda8be4137beae728786a7731d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:46 UTC) from {username='slave'} at 127.0.0.1:45194
I20240502 21:26:46.670890 2328 ts_tablet_manager.cc:1875] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:46.671356 2329 tablet_replica.cc:330] T 594137cda8be4137beae728786a7731d P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:46.671653 2329 raft_consensus.cc:2229] T 594137cda8be4137beae728786a7731d P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:46.671918 2329 raft_consensus.cc:2258] T 594137cda8be4137beae728786a7731d P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:46.675318 2329 ts_tablet_manager.cc:1875] T 594137cda8be4137beae728786a7731d P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:46.679872 2328 ts_tablet_manager.cc:1888] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:46.680049 2328 log.cc:1196] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/594137cda8be4137beae728786a7731d
I20240502 21:26:46.680567 2328 ts_tablet_manager.cc:1909] T 594137cda8be4137beae728786a7731d P 2d50f33557944bb480ea32199d9ee4f1: Deleting consensus metadata
I20240502 21:26:46.681658 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet 594137cda8be4137beae728786a7731d (table rebalancer_dynamic_table_new_name [id=8123e6cbc25b44e1899a876d634dfaa5]) successfully deleted
I20240502 21:26:46.685564 2329 ts_tablet_manager.cc:1888] T 594137cda8be4137beae728786a7731d P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:46.686378 2329 log.cc:1196] T 594137cda8be4137beae728786a7731d P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/594137cda8be4137beae728786a7731d
I20240502 21:26:46.686901 2329 ts_tablet_manager.cc:1909] T 594137cda8be4137beae728786a7731d P a44f41b0b6644fb6aa29476aed8c964f: Deleting consensus metadata
I20240502 21:26:46.688031 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet 594137cda8be4137beae728786a7731d (table rebalancer_dynamic_table_new_name [id=8123e6cbc25b44e1899a876d634dfaa5]) successfully deleted
I20240502 21:26:46.693825 31393 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_extra_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:46.696094 1946 tablet_service.cc:1514] Processing DeleteTablet for tablet 594137cda8be4137beae728786a7731d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:46 UTC) from {username='slave'} at 127.0.0.1:33002
I20240502 21:26:46.696729 2892 tablet_replica.cc:330] T 594137cda8be4137beae728786a7731d P 9dec645b473f40578e726f3e50e219bf: stopping tablet replica
I20240502 21:26:46.696950 2892 raft_consensus.cc:2229] T 594137cda8be4137beae728786a7731d P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:46.697217 2892 raft_consensus.cc:2258] T 594137cda8be4137beae728786a7731d P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:46.697762 2515 raft_consensus.cc:2932] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:46.699124 31797 consensus_queue.cc:237] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:26:46.700536 2892 ts_tablet_manager.cc:1875] T 594137cda8be4137beae728786a7731d P 9dec645b473f40578e726f3e50e219bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:46.708698 2210 tablet_service.cc:1467] Processing CreateTablet for tablet f5e82c0ab5c441efbcfbcd383b8db82f (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=4cfb4e8589a24f279c0e7f552012212c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:46.709172 2210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5e82c0ab5c441efbcfbcd383b8db82f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:46.712297 2075 tablet_service.cc:1467] Processing CreateTablet for tablet f5e82c0ab5c441efbcfbcd383b8db82f (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=4cfb4e8589a24f279c0e7f552012212c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:46.712425 31666 raft_consensus.cc:1261] T f9bbc56ba2f841b9bb36a864e1a1f808 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:46.712956 1811 tablet_service.cc:1467] Processing CreateTablet for tablet f5e82c0ab5c441efbcfbcd383b8db82f (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=4cfb4e8589a24f279c0e7f552012212c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:46.713145 31926 raft_consensus.cc:1261] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:46.713532 1811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5e82c0ab5c441efbcfbcd383b8db82f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:46.713851 2799 consensus_queue.cc:1035] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:46.714406 2799 consensus_queue.cc:1035] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:46.717859 2075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5e82c0ab5c441efbcfbcd383b8db82f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:46.725520 2076 tablet_service.cc:1467] Processing CreateTablet for tablet 46279270b2334633906af57c4b43e969 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=4cfb4e8589a24f279c0e7f552012212c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:46.725833 2076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46279270b2334633906af57c4b43e969. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:46.726569 2898 tablet_bootstrap.cc:492] T f5e82c0ab5c441efbcfbcd383b8db82f P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:46.728305 2898 tablet_bootstrap.cc:654] T f5e82c0ab5c441efbcfbcd383b8db82f P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:46.730126 2799 raft_consensus.cc:2932] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:46.731585 2209 tablet_service.cc:1467] Processing CreateTablet for tablet 46279270b2334633906af57c4b43e969 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=4cfb4e8589a24f279c0e7f552012212c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:46.732044 2209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46279270b2334633906af57c4b43e969. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:46.734238 1812 tablet_service.cc:1467] Processing CreateTablet for tablet 4f09394db52e489e9bc42a4883ee8b1c (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=4cfb4e8589a24f279c0e7f552012212c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:46.734655 1812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f09394db52e489e9bc42a4883ee8b1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:46.736748 2077 tablet_service.cc:1467] Processing CreateTablet for tablet 4f09394db52e489e9bc42a4883ee8b1c (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=4cfb4e8589a24f279c0e7f552012212c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:46.737077 2077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f09394db52e489e9bc42a4883ee8b1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:46.737279 31905 tablet_service.cc:1467] Processing CreateTablet for tablet 4f09394db52e489e9bc42a4883ee8b1c (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=4cfb4e8589a24f279c0e7f552012212c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:46.737756 31905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f09394db52e489e9bc42a4883ee8b1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:46.748883 31393 catalog_manager.cc:5484] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:46.749647 2096 raft_consensus.cc:2932] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:46.731114 31666 raft_consensus.cc:2932] T f9bbc56ba2f841b9bb36a864e1a1f808 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
W20240502 21:26:46.755542 31730 consensus_peers.cc:485] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f9bbc56ba2f841b9bb36a864e1a1f808. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:46.755741 31817 tablet_copy_service.cc:140] P 08d7a0680d244b55b93d17f9ab57b84c: Received BeginTabletCopySession request for tablet b407c2c9b3a14ce3ada52b7856c6ecee from peer 9dec645b473f40578e726f3e50e219bf ({username='slave'} at 127.24.226.134:42809)
I20240502 21:26:46.755714 1946 tablet_service.cc:1467] Processing CreateTablet for tablet 46279270b2334633906af57c4b43e969 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=4cfb4e8589a24f279c0e7f552012212c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:46.755967 31817 tablet_copy_service.cc:161] P 08d7a0680d244b55b93d17f9ab57b84c: Beginning new tablet copy session on tablet b407c2c9b3a14ce3ada52b7856c6ecee from peer 9dec645b473f40578e726f3e50e219bf at {username='slave'} at 127.24.226.134:42809: session id = 9dec645b473f40578e726f3e50e219bf-b407c2c9b3a14ce3ada52b7856c6ecee
I20240502 21:26:46.756167 1946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46279270b2334633906af57c4b43e969. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:46.759474 31817 tablet_copy_source_session.cc:205] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:46.760797 2228 raft_consensus.cc:1261] T f02f83ec0a494da5b66241d9b379a03f P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:46.761307 2792 consensus_queue.cc:1035] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:46.761441 31535 raft_consensus.cc:2932] T f02f83ec0a494da5b66241d9b379a03f P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:46.765578 2892 ts_tablet_manager.cc:1888] T 594137cda8be4137beae728786a7731d P 9dec645b473f40578e726f3e50e219bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:46.767940 2901 tablet_bootstrap.cc:492] T f5e82c0ab5c441efbcfbcd383b8db82f P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:46.770102 2901 tablet_bootstrap.cc:654] T f5e82c0ab5c441efbcfbcd383b8db82f P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:46.767838 2872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b407c2c9b3a14ce3ada52b7856c6ecee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:46.771884 31394 catalog_manager.cc:5484] T f02f83ec0a494da5b66241d9b379a03f P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: config changed from index 2 to 3, a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9dec645b473f40578e726f3e50e219bf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:46.775789 31391 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:26:46.775789 31395 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240502 21:26:46.777722 31395 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240502 21:26:46.783075 31395 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 3d1bfb337f3c47a78c742cb665d27ffb
I20240502 21:26:46.783721 31395 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 4 replicas of tablet fe4e41225da24d7c91c6bf7733a5e067
I20240502 21:26:46.784093 31395 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 80023f3b254c4369954ab92a085a0ec5
I20240502 21:26:46.784624 31647 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d1bfb337f3c47a78c742cb665d27ffb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:46 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:46.784641 31775 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d1bfb337f3c47a78c742cb665d27ffb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:46 UTC) from {username='slave'} at 127.0.0.1:51596
I20240502 21:26:46.784816 31774 tablet_service.cc:1514] Processing DeleteTablet for tablet 80023f3b254c4369954ab92a085a0ec5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:46 UTC) from {username='slave'} at 127.0.0.1:51596
I20240502 21:26:46.784852 31645 tablet_service.cc:1514] Processing DeleteTablet for tablet fe4e41225da24d7c91c6bf7733a5e067 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:46 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:46.784968 2602 tablet_replica.cc:330] T 3d1bfb337f3c47a78c742cb665d27ffb P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:46.785137 2602 raft_consensus.cc:2229] T 3d1bfb337f3c47a78c742cb665d27ffb P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:46.785343 31515 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d1bfb337f3c47a78c742cb665d27ffb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:46 UTC) from {username='slave'} at 127.0.0.1:45194
I20240502 21:26:46.785579 31906 tablet_service.cc:1514] Processing DeleteTablet for tablet fe4e41225da24d7c91c6bf7733a5e067 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:46 UTC) from {username='slave'} at 127.0.0.1:47824
I20240502 21:26:46.785352 31516 tablet_service.cc:1514] Processing DeleteTablet for tablet fe4e41225da24d7c91c6bf7733a5e067 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:46 UTC) from {username='slave'} at 127.0.0.1:45194
I20240502 21:26:46.785800 2329 tablet_replica.cc:330] T 3d1bfb337f3c47a78c742cb665d27ffb P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:46.785842 2328 tablet_replica.cc:330] T fe4e41225da24d7c91c6bf7733a5e067 P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:26:46.785895 2075 tablet_service.cc:1514] Processing DeleteTablet for tablet fe4e41225da24d7c91c6bf7733a5e067 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:46 UTC) from {username='slave'} at 127.0.0.1:36740
I20240502 21:26:46.785951 2329 raft_consensus.cc:2229] T 3d1bfb337f3c47a78c742cb665d27ffb P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:46.786028 2328 raft_consensus.cc:2229] T fe4e41225da24d7c91c6bf7733a5e067 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:46.786228 2329 raft_consensus.cc:2258] T 3d1bfb337f3c47a78c742cb665d27ffb P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:46.786198 2872 tablet_copy_client.cc:783] T b407c2c9b3a14ce3ada52b7856c6ecee P 9dec645b473f40578e726f3e50e219bf: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:46.786298 2328 raft_consensus.cc:2258] T fe4e41225da24d7c91c6bf7733a5e067 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:46.786477 2872 tablet_copy_client.cc:647] T b407c2c9b3a14ce3ada52b7856c6ecee P 9dec645b473f40578e726f3e50e219bf: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:46.786885 2329 ts_tablet_manager.cc:1875] T 3d1bfb337f3c47a78c742cb665d27ffb P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:46.787925 31645 tablet_service.cc:1514] Processing DeleteTablet for tablet 80023f3b254c4369954ab92a085a0ec5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:46 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:46.789203 31906 tablet_service.cc:1514] Processing DeleteTablet for tablet 80023f3b254c4369954ab92a085a0ec5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:46 UTC) from {username='slave'} at 127.0.0.1:47824
I20240502 21:26:46.790880 2328 ts_tablet_manager.cc:1875] T fe4e41225da24d7c91c6bf7733a5e067 P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:46.805918 2892 log.cc:1196] T 594137cda8be4137beae728786a7731d P 9dec645b473f40578e726f3e50e219bf: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal/wals/594137cda8be4137beae728786a7731d
I20240502 21:26:46.806468 2892 ts_tablet_manager.cc:1909] T 594137cda8be4137beae728786a7731d P 9dec645b473f40578e726f3e50e219bf: Deleting consensus metadata
I20240502 21:26:46.812222 2909 ts_tablet_manager.cc:902] T fe4e41225da24d7c91c6bf7733a5e067 P 1ade0354c44c4c37ae480c7a94b11e1b: Initiating tablet copy from peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:46.812706 31926 raft_consensus.cc:2932] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:46.814220 2328 ts_tablet_manager.cc:1888] T fe4e41225da24d7c91c6bf7733a5e067 P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:46.814342 2328 log.cc:1196] T fe4e41225da24d7c91c6bf7733a5e067 P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/fe4e41225da24d7c91c6bf7733a5e067
W20240502 21:26:46.814416 31381 catalog_manager.cc:4797] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): delete failed for tablet fe4e41225da24d7c91c6bf7733a5e067 because the tablet was not found. No further retry: Not found: Tablet not found: fe4e41225da24d7c91c6bf7733a5e067
I20240502 21:26:46.814801 2328 ts_tablet_manager.cc:1909] T fe4e41225da24d7c91c6bf7733a5e067 P 2d50f33557944bb480ea32199d9ee4f1: Deleting consensus metadata
I20240502 21:26:46.815826 2328 tablet_replica.cc:330] T 80023f3b254c4369954ab92a085a0ec5 P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:26:46.815939 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet fe4e41225da24d7c91c6bf7733a5e067 (table rebalance_test_table_16 [id=ba6507441fb04f29a519bb81d3dba0e1]) successfully deleted
I20240502 21:26:46.816087 2328 raft_consensus.cc:2229] T 80023f3b254c4369954ab92a085a0ec5 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:46.816421 2328 raft_consensus.cc:2258] T 80023f3b254c4369954ab92a085a0ec5 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:46.817055 31379 catalog_manager.cc:4830] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): tablet 594137cda8be4137beae728786a7731d (table rebalancer_dynamic_table_new_name [id=8123e6cbc25b44e1899a876d634dfaa5]) successfully deleted
I20240502 21:26:46.817225 2328 ts_tablet_manager.cc:1875] T 80023f3b254c4369954ab92a085a0ec5 P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:46.818477 1966 consensus_queue.cc:237] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } }
I20240502 21:26:46.821041 2329 ts_tablet_manager.cc:1888] T 3d1bfb337f3c47a78c742cb665d27ffb P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:46.821242 2329 log.cc:1196] T 3d1bfb337f3c47a78c742cb665d27ffb P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/3d1bfb337f3c47a78c742cb665d27ffb
I20240502 21:26:46.821758 2329 ts_tablet_manager.cc:1909] T 3d1bfb337f3c47a78c742cb665d27ffb P a44f41b0b6644fb6aa29476aed8c964f: Deleting consensus metadata
I20240502 21:26:46.822052 31535 raft_consensus.cc:1261] T f02f83ec0a494da5b66241d9b379a03f P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:46.822973 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet 3d1bfb337f3c47a78c742cb665d27ffb (table rebalance_test_table_16 [id=ba6507441fb04f29a519bb81d3dba0e1]) successfully deleted
I20240502 21:26:46.822984 2792 consensus_queue.cc:1035] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:46.823244 2329 tablet_replica.cc:330] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:46.823649 2329 raft_consensus.cc:2229] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:46.825838 2894 raft_consensus.cc:2932] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:46.827292 2901 tablet_bootstrap.cc:492] T f5e82c0ab5c441efbcfbcd383b8db82f P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:46.827441 2901 ts_tablet_manager.cc:1367] T f5e82c0ab5c441efbcfbcd383b8db82f P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.060s user 0.003s sys 0.003s
I20240502 21:26:46.828105 2901 raft_consensus.cc:348] T f5e82c0ab5c441efbcfbcd383b8db82f P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:46.828272 2901 raft_consensus.cc:374] T f5e82c0ab5c441efbcfbcd383b8db82f P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:46.828339 2901 raft_consensus.cc:729] T f5e82c0ab5c441efbcfbcd383b8db82f P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:46.831701 2901 consensus_queue.cc:260] T f5e82c0ab5c441efbcfbcd383b8db82f P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:46.832335 31535 raft_consensus.cc:2932] T f02f83ec0a494da5b66241d9b379a03f P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:46.833990 2901 ts_tablet_manager.cc:1398] T f5e82c0ab5c441efbcfbcd383b8db82f P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 21:26:46.834045 2329 raft_consensus.cc:2258] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:46.835042 2901 tablet_bootstrap.cc:492] T 4f09394db52e489e9bc42a4883ee8b1c P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:46.835247 2329 ts_tablet_manager.cc:1875] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:46.839653 2901 tablet_bootstrap.cc:654] T 4f09394db52e489e9bc42a4883ee8b1c P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:46.840773 2602 raft_consensus.cc:2258] T 3d1bfb337f3c47a78c742cb665d27ffb P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:46.841485 2602 ts_tablet_manager.cc:1875] T 3d1bfb337f3c47a78c742cb665d27ffb P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:46.841529 2328 ts_tablet_manager.cc:1888] T 80023f3b254c4369954ab92a085a0ec5 P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:46.841641 2328 log.cc:1196] T 80023f3b254c4369954ab92a085a0ec5 P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/80023f3b254c4369954ab92a085a0ec5
I20240502 21:26:46.842034 2328 ts_tablet_manager.cc:1909] T 80023f3b254c4369954ab92a085a0ec5 P 2d50f33557944bb480ea32199d9ee4f1: Deleting consensus metadata
I20240502 21:26:46.844231 31379 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f02f83ec0a494da5b66241d9b379a03f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:46.844594 2096 raft_consensus.cc:1261] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:46.845536 2872 tablet_copy_client.cc:523] T b407c2c9b3a14ce3ada52b7856c6ecee P 9dec645b473f40578e726f3e50e219bf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:46.848583 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet 80023f3b254c4369954ab92a085a0ec5 (table rebalance_test_table_16 [id=ba6507441fb04f29a519bb81d3dba0e1]) successfully deleted
I20240502 21:26:46.850018 2872 tablet_bootstrap.cc:492] T b407c2c9b3a14ce3ada52b7856c6ecee P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:46.851953 2602 ts_tablet_manager.cc:1888] T 3d1bfb337f3c47a78c742cb665d27ffb P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:46.852114 2602 log.cc:1196] T 3d1bfb337f3c47a78c742cb665d27ffb P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/3d1bfb337f3c47a78c742cb665d27ffb
I20240502 21:26:46.852720 2602 ts_tablet_manager.cc:1909] T 3d1bfb337f3c47a78c742cb665d27ffb P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:26:46.853960 2602 tablet_replica.cc:330] T fe4e41225da24d7c91c6bf7733a5e067 P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:46.854115 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet 3d1bfb337f3c47a78c742cb665d27ffb (table rebalance_test_table_16 [id=ba6507441fb04f29a519bb81d3dba0e1]) successfully deleted
I20240502 21:26:46.854256 2602 raft_consensus.cc:2229] T fe4e41225da24d7c91c6bf7733a5e067 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:46.854610 2602 raft_consensus.cc:2258] T fe4e41225da24d7c91c6bf7733a5e067 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:46.856249 2904 tablet_bootstrap.cc:492] T 4f09394db52e489e9bc42a4883ee8b1c P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:46.858510 2904 tablet_bootstrap.cc:654] T 4f09394db52e489e9bc42a4883ee8b1c P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:46.860641 2907 tablet_replica.cc:330] T 3d1bfb337f3c47a78c742cb665d27ffb P 08d7a0680d244b55b93d17f9ab57b84c: stopping tablet replica
I20240502 21:26:46.860971 2907 raft_consensus.cc:2229] T 3d1bfb337f3c47a78c742cb665d27ffb P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:46.861354 2907 raft_consensus.cc:2258] T 3d1bfb337f3c47a78c742cb665d27ffb P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:46.862226 2907 ts_tablet_manager.cc:1875] T 3d1bfb337f3c47a78c742cb665d27ffb P 08d7a0680d244b55b93d17f9ab57b84c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:46.866591 2901 tablet_bootstrap.cc:492] T 4f09394db52e489e9bc42a4883ee8b1c P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:46.867166 2902 tablet_bootstrap.cc:492] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:46.867713 2602 ts_tablet_manager.cc:1875] T fe4e41225da24d7c91c6bf7733a5e067 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:46.868605 2901 ts_tablet_manager.cc:1367] T 4f09394db52e489e9bc42a4883ee8b1c P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.034s user 0.001s sys 0.007s
I20240502 21:26:46.869454 2901 raft_consensus.cc:348] T 4f09394db52e489e9bc42a4883ee8b1c P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:46.871081 2901 raft_consensus.cc:374] T 4f09394db52e489e9bc42a4883ee8b1c P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:46.872015 2901 raft_consensus.cc:729] T 4f09394db52e489e9bc42a4883ee8b1c P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:46.872181 31390 catalog_manager.cc:5484] T f02f83ec0a494da5b66241d9b379a03f P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: config changed from index 3 to 4, VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) evicted. New cstate: current_term: 1 leader_uuid: "9dec645b473f40578e726f3e50e219bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:46.872413 2901 consensus_queue.cc:260] T 4f09394db52e489e9bc42a4883ee8b1c P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:46.873531 2901 ts_tablet_manager.cc:1398] T 4f09394db52e489e9bc42a4883ee8b1c P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 21:26:46.879827 2902 tablet_bootstrap.cc:654] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:46.880388 2602 ts_tablet_manager.cc:1888] T fe4e41225da24d7c91c6bf7733a5e067 P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:46.880812 2602 log.cc:1196] T fe4e41225da24d7c91c6bf7733a5e067 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/fe4e41225da24d7c91c6bf7733a5e067
I20240502 21:26:46.881271 2602 ts_tablet_manager.cc:1909] T fe4e41225da24d7c91c6bf7733a5e067 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:26:46.881713 2792 consensus_queue.cc:1035] T f02f83ec0a494da5b66241d9b379a03f P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:46.884356 2075 tablet_service.cc:1467] Processing CreateTablet for tablet f332ab12c37f459ca85585657a0fe01e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b6e29d3838874defa6668514ac1c2423]), partition=
I20240502 21:26:46.884779 2075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f332ab12c37f459ca85585657a0fe01e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 21:26:46.886080 31379 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet f02f83ec0a494da5b66241d9b379a03f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 3 but the committed config has opid_index of 4
I20240502 21:26:46.887068 1945 tablet_service.cc:1467] Processing CreateTablet for tablet f332ab12c37f459ca85585657a0fe01e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b6e29d3838874defa6668514ac1c2423]), partition=
I20240502 21:26:46.887470 1945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f332ab12c37f459ca85585657a0fe01e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:46.890362 2096 raft_consensus.cc:2932] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:46.892199 2329 ts_tablet_manager.cc:1888] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:46.892366 2329 log.cc:1196] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/fe4e41225da24d7c91c6bf7733a5e067
I20240502 21:26:46.893676 2902 tablet_bootstrap.cc:492] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:46.893882 2902 ts_tablet_manager.cc:1367] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.002s
I20240502 21:26:46.894748 2209 tablet_service.cc:1514] Processing DeleteTablet for tablet f02f83ec0a494da5b66241d9b379a03f with delete_type TABLET_DATA_TOMBSTONED (TS d1406e92aec84ac585304e57bd37a40d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36984
I20240502 21:26:46.894889 2902 raft_consensus.cc:348] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:46.895248 2902 raft_consensus.cc:374] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:46.895416 2902 raft_consensus.cc:729] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:46.896137 2902 consensus_queue.cc:260] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:46.897768 2902 ts_tablet_manager.cc:1398] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 21:26:46.898252 2902 tablet_bootstrap.cc:492] T 46279270b2334633906af57c4b43e969 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:46.900027 2904 tablet_bootstrap.cc:492] T 4f09394db52e489e9bc42a4883ee8b1c P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:46.900224 2904 ts_tablet_manager.cc:1367] T 4f09394db52e489e9bc42a4883ee8b1c P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.000s
I20240502 21:26:46.901010 2904 raft_consensus.cc:348] T 4f09394db52e489e9bc42a4883ee8b1c P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:46.901296 2904 raft_consensus.cc:374] T 4f09394db52e489e9bc42a4883ee8b1c P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:46.901458 2904 raft_consensus.cc:729] T 4f09394db52e489e9bc42a4883ee8b1c P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:46.901732 2904 consensus_queue.cc:260] T 4f09394db52e489e9bc42a4883ee8b1c P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:46.895856 2210 tablet_service.cc:1467] Processing CreateTablet for tablet f332ab12c37f459ca85585657a0fe01e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b6e29d3838874defa6668514ac1c2423]), partition=
I20240502 21:26:46.903057 2210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f332ab12c37f459ca85585657a0fe01e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:46.903360 2904 ts_tablet_manager.cc:1398] T 4f09394db52e489e9bc42a4883ee8b1c P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 21:26:46.903529 2329 ts_tablet_manager.cc:1909] T fe4e41225da24d7c91c6bf7733a5e067 P a44f41b0b6644fb6aa29476aed8c964f: Deleting consensus metadata
I20240502 21:26:46.904961 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet fe4e41225da24d7c91c6bf7733a5e067 (table rebalance_test_table_16 [id=ba6507441fb04f29a519bb81d3dba0e1]) successfully deleted
I20240502 21:26:46.905119 2902 tablet_bootstrap.cc:654] T 46279270b2334633906af57c4b43e969 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:46.903412 2907 ts_tablet_manager.cc:1888] T 3d1bfb337f3c47a78c742cb665d27ffb P 08d7a0680d244b55b93d17f9ab57b84c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:46.905953 2907 log.cc:1196] T 3d1bfb337f3c47a78c742cb665d27ffb P 08d7a0680d244b55b93d17f9ab57b84c: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/wals/3d1bfb337f3c47a78c742cb665d27ffb
I20240502 21:26:46.906157 2898 tablet_bootstrap.cc:492] T f5e82c0ab5c441efbcfbcd383b8db82f P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:46.906611 2907 ts_tablet_manager.cc:1909] T 3d1bfb337f3c47a78c742cb665d27ffb P 08d7a0680d244b55b93d17f9ab57b84c: Deleting consensus metadata
I20240502 21:26:46.907454 2898 ts_tablet_manager.cc:1367] T f5e82c0ab5c441efbcfbcd383b8db82f P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.181s user 0.044s sys 0.048s
I20240502 21:26:46.908154 2898 raft_consensus.cc:348] T f5e82c0ab5c441efbcfbcd383b8db82f P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:46.908370 2898 raft_consensus.cc:374] T f5e82c0ab5c441efbcfbcd383b8db82f P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:46.908492 2898 raft_consensus.cc:729] T f5e82c0ab5c441efbcfbcd383b8db82f P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:46.909078 2898 consensus_queue.cc:260] T f5e82c0ab5c441efbcfbcd383b8db82f P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:46.909248 31381 catalog_manager.cc:4830] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): tablet 3d1bfb337f3c47a78c742cb665d27ffb (table rebalance_test_table_16 [id=ba6507441fb04f29a519bb81d3dba0e1]) successfully deleted
I20240502 21:26:46.909840 2898 ts_tablet_manager.cc:1398] T f5e82c0ab5c441efbcfbcd383b8db82f P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:46.909955 2907 tablet_replica.cc:330] T 80023f3b254c4369954ab92a085a0ec5 P 08d7a0680d244b55b93d17f9ab57b84c: stopping tablet replica
I20240502 21:26:46.910184 2898 tablet_bootstrap.cc:492] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:46.910209 2907 raft_consensus.cc:2229] T 80023f3b254c4369954ab92a085a0ec5 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:46.910660 2907 raft_consensus.cc:2258] T 80023f3b254c4369954ab92a085a0ec5 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:46.911698 2898 tablet_bootstrap.cc:654] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:46.911957 2907 ts_tablet_manager.cc:1875] T 80023f3b254c4369954ab92a085a0ec5 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:46.920030 2602 tablet_replica.cc:330] T 80023f3b254c4369954ab92a085a0ec5 P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:46.920253 2602 raft_consensus.cc:2229] T 80023f3b254c4369954ab92a085a0ec5 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:46.920775 2602 raft_consensus.cc:2258] T 80023f3b254c4369954ab92a085a0ec5 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:46.921531 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet fe4e41225da24d7c91c6bf7733a5e067 (table rebalance_test_table_16 [id=ba6507441fb04f29a519bb81d3dba0e1]) successfully deleted
I20240502 21:26:46.921952 2602 ts_tablet_manager.cc:1875] T 80023f3b254c4369954ab92a085a0ec5 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:46.924571 2902 tablet_bootstrap.cc:492] T 46279270b2334633906af57c4b43e969 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:46.924772 2902 ts_tablet_manager.cc:1367] T 46279270b2334633906af57c4b43e969 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.001s
I20240502 21:26:46.925401 2902 raft_consensus.cc:348] T 46279270b2334633906af57c4b43e969 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:46.925813 2902 raft_consensus.cc:374] T 46279270b2334633906af57c4b43e969 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:46.925916 2902 raft_consensus.cc:729] T 46279270b2334633906af57c4b43e969 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:46.926295 2902 consensus_queue.cc:260] T 46279270b2334633906af57c4b43e969 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:46.928742 2902 ts_tablet_manager.cc:1398] T 46279270b2334633906af57c4b43e969 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 21:26:46.928869 2602 ts_tablet_manager.cc:1888] T 80023f3b254c4369954ab92a085a0ec5 P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:46.929073 2602 log.cc:1196] T 80023f3b254c4369954ab92a085a0ec5 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/80023f3b254c4369954ab92a085a0ec5
I20240502 21:26:46.929307 2910 tablet_bootstrap.cc:492] T 46279270b2334633906af57c4b43e969 P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:46.929807 2902 tablet_bootstrap.cc:492] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:46.931066 2902 tablet_bootstrap.cc:654] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:46.931267 2910 tablet_bootstrap.cc:654] T 46279270b2334633906af57c4b43e969 P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:46.929592 2602 ts_tablet_manager.cc:1909] T 80023f3b254c4369954ab92a085a0ec5 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:26:46.932215 2898 tablet_bootstrap.cc:492] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:46.932360 2898 ts_tablet_manager.cc:1367] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.003s
I20240502 21:26:46.932590 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet 80023f3b254c4369954ab92a085a0ec5 (table rebalance_test_table_16 [id=ba6507441fb04f29a519bb81d3dba0e1]) successfully deleted
I20240502 21:26:46.932974 2907 ts_tablet_manager.cc:1888] T 80023f3b254c4369954ab92a085a0ec5 P 08d7a0680d244b55b93d17f9ab57b84c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:46.933121 2907 log.cc:1196] T 80023f3b254c4369954ab92a085a0ec5 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/wals/80023f3b254c4369954ab92a085a0ec5
I20240502 21:26:46.933060 2898 raft_consensus.cc:348] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:46.933279 2898 raft_consensus.cc:374] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:46.933446 2898 raft_consensus.cc:729] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:46.933638 2907 ts_tablet_manager.cc:1909] T 80023f3b254c4369954ab92a085a0ec5 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting consensus metadata
I20240502 21:26:46.933804 2898 consensus_queue.cc:260] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:46.934440 2898 ts_tablet_manager.cc:1398] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:46.934753 2898 tablet_bootstrap.cc:492] T 4f09394db52e489e9bc42a4883ee8b1c P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:46.934857 31381 catalog_manager.cc:4830] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): tablet 80023f3b254c4369954ab92a085a0ec5 (table rebalance_test_table_16 [id=ba6507441fb04f29a519bb81d3dba0e1]) successfully deleted
I20240502 21:26:46.933615 2228 raft_consensus.cc:2932] T f02f83ec0a494da5b66241d9b379a03f P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:46.936250 2898 tablet_bootstrap.cc:654] T 4f09394db52e489e9bc42a4883ee8b1c P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:46.941782 2898 tablet_bootstrap.cc:492] T 4f09394db52e489e9bc42a4883ee8b1c P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:46.947340 2898 ts_tablet_manager.cc:1367] T 4f09394db52e489e9bc42a4883ee8b1c P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 21:26:46.947665 2902 tablet_bootstrap.cc:492] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:46.947799 2902 ts_tablet_manager.cc:1367] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.007s
I20240502 21:26:46.948062 2898 raft_consensus.cc:348] T 4f09394db52e489e9bc42a4883ee8b1c P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:46.948248 2898 raft_consensus.cc:374] T 4f09394db52e489e9bc42a4883ee8b1c P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:46.948341 2898 raft_consensus.cc:729] T 4f09394db52e489e9bc42a4883ee8b1c P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:46.948608 2898 consensus_queue.cc:260] T 4f09394db52e489e9bc42a4883ee8b1c P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:46.948629 2902 raft_consensus.cc:348] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:46.948889 2902 raft_consensus.cc:374] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:46.949003 2902 raft_consensus.cc:729] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:46.949235 2902 consensus_queue.cc:260] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:46.949822 2902 ts_tablet_manager.cc:1398] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:46.950031 2898 ts_tablet_manager.cc:1398] T 4f09394db52e489e9bc42a4883ee8b1c P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:46.950276 2898 tablet_bootstrap.cc:492] T f332ab12c37f459ca85585657a0fe01e P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:46.952679 2898 tablet_bootstrap.cc:654] T f332ab12c37f459ca85585657a0fe01e P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:46.957839 2931 ts_tablet_manager.cc:902] T b9342dbe0497499ba4c238f2ed033293 P 05f4fac41c4f4959b9b316cb112d0ada: Initiating tablet copy from peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:26:46.958160 2910 tablet_bootstrap.cc:492] T 46279270b2334633906af57c4b43e969 P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:26:46.958346 2910 ts_tablet_manager.cc:1367] T 46279270b2334633906af57c4b43e969 P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20240502 21:26:46.959142 2910 raft_consensus.cc:348] T 46279270b2334633906af57c4b43e969 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:46.959471 2910 raft_consensus.cc:374] T 46279270b2334633906af57c4b43e969 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:46.959635 2910 raft_consensus.cc:729] T 46279270b2334633906af57c4b43e969 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:26:46.960012 2910 consensus_queue.cc:260] T 46279270b2334633906af57c4b43e969 P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:46.961593 2816 raft_consensus.cc:482] T 4f09394db52e489e9bc42a4883ee8b1c P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:46.961691 2910 ts_tablet_manager.cc:1398] T 46279270b2334633906af57c4b43e969 P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:46.961755 2816 raft_consensus.cc:504] T 4f09394db52e489e9bc42a4883ee8b1c P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:46.962462 2816 leader_election.cc:288] T 4f09394db52e489e9bc42a4883ee8b1c P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:46.963363 2917 tablet_replica.cc:330] T f02f83ec0a494da5b66241d9b379a03f P d1406e92aec84ac585304e57bd37a40d: stopping tablet replica
I20240502 21:26:46.963550 2917 raft_consensus.cc:2229] T f02f83ec0a494da5b66241d9b379a03f P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:46.963794 2917 raft_consensus.cc:2258] T f02f83ec0a494da5b66241d9b379a03f P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:46.963768 2096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f09394db52e489e9bc42a4883ee8b1c" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:46.963954 2096 raft_consensus.cc:2454] T 4f09394db52e489e9bc42a4883ee8b1c P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 0.
I20240502 21:26:46.964794 1769 leader_election.cc:302] T 4f09394db52e489e9bc42a4883ee8b1c P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 1e04157b98dd4fad98858c290b73c3db; no voters:
I20240502 21:26:46.966523 2816 raft_consensus.cc:2781] T 4f09394db52e489e9bc42a4883ee8b1c P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:46.966606 2816 raft_consensus.cc:482] T 4f09394db52e489e9bc42a4883ee8b1c P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:46.966712 2816 raft_consensus.cc:3037] T 4f09394db52e489e9bc42a4883ee8b1c P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:46.969214 2816 raft_consensus.cc:504] T 4f09394db52e489e9bc42a4883ee8b1c P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:46.970135 2917 ts_tablet_manager.cc:1875] T f02f83ec0a494da5b66241d9b379a03f P d1406e92aec84ac585304e57bd37a40d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:46.977228 2096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f09394db52e489e9bc42a4883ee8b1c" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:26:46.977398 2096 raft_consensus.cc:3037] T 4f09394db52e489e9bc42a4883ee8b1c P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:46.979805 2096 raft_consensus.cc:2454] T 4f09394db52e489e9bc42a4883ee8b1c P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:26:46.980783 1769 leader_election.cc:302] T 4f09394db52e489e9bc42a4883ee8b1c P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 1e04157b98dd4fad98858c290b73c3db; no voters:
I20240502 21:26:46.981065 2763 raft_consensus.cc:2781] T 4f09394db52e489e9bc42a4883ee8b1c P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:46.981184 2763 raft_consensus.cc:686] T 4f09394db52e489e9bc42a4883ee8b1c P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Becoming Leader. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Running, Role: LEADER
I20240502 21:26:46.981423 2763 consensus_queue.cc:237] T 4f09394db52e489e9bc42a4883ee8b1c P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:46.984884 2816 leader_election.cc:288] T 4f09394db52e489e9bc42a4883ee8b1c P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Requested vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:46.993476 2917 ts_tablet_manager.cc:1888] T f02f83ec0a494da5b66241d9b379a03f P d1406e92aec84ac585304e57bd37a40d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:26:46.993687 2917 log.cc:1196] T f02f83ec0a494da5b66241d9b379a03f P d1406e92aec84ac585304e57bd37a40d: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal/wals/f02f83ec0a494da5b66241d9b379a03f
I20240502 21:26:46.994716 31381 catalog_manager.cc:4830] TS d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): tablet f02f83ec0a494da5b66241d9b379a03f (table rebalancer_extra_table_2 [id=408b4cadaf474250a5666b6a49f4f8cc]) successfully deleted
I20240502 21:26:46.997357 2909 tablet_copy_client.cc:308] T fe4e41225da24d7c91c6bf7733a5e067 P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.129:41039
I20240502 21:26:46.998546 2891 raft_consensus.cc:482] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:46.998713 2891 raft_consensus.cc:504] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:46.999580 2891 leader_election.cc:288] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:26:47.000384 2096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e82c0ab5c441efbcfbcd383b8db82f" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:47.000669 2096 raft_consensus.cc:2454] T f5e82c0ab5c441efbcfbcd383b8db82f P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:26:47.000772 1833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e82c0ab5c441efbcfbcd383b8db82f" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:26:47.000975 1833 raft_consensus.cc:2454] T f5e82c0ab5c441efbcfbcd383b8db82f P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:26:47.001159 2163 leader_election.cc:302] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:47.001652 2891 raft_consensus.cc:2781] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:47.001749 2891 raft_consensus.cc:482] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:47.001822 2891 raft_consensus.cc:3037] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:47.004274 2891 raft_consensus.cc:504] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:47.004715 2872 tablet_bootstrap.cc:492] T b407c2c9b3a14ce3ada52b7856c6ecee P 9dec645b473f40578e726f3e50e219bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:47.005069 2872 tablet_bootstrap.cc:492] T b407c2c9b3a14ce3ada52b7856c6ecee P 9dec645b473f40578e726f3e50e219bf: Bootstrap complete.
I20240502 21:26:47.005288 2872 ts_tablet_manager.cc:1367] T b407c2c9b3a14ce3ada52b7856c6ecee P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.155s user 0.053s sys 0.005s
I20240502 21:26:47.006035 2872 raft_consensus.cc:348] T b407c2c9b3a14ce3ada52b7856c6ecee P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:26:47.006233 2872 raft_consensus.cc:729] T b407c2c9b3a14ce3ada52b7856c6ecee P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: LEARNER
I20240502 21:26:47.006412 2872 consensus_queue.cc:260] T b407c2c9b3a14ce3ada52b7856c6ecee P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:26:47.007427 2872 ts_tablet_manager.cc:1398] T b407c2c9b3a14ce3ada52b7856c6ecee P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:47.008651 31817 tablet_copy_service.cc:342] P 08d7a0680d244b55b93d17f9ab57b84c: Request end of tablet copy session 9dec645b473f40578e726f3e50e219bf-b407c2c9b3a14ce3ada52b7856c6ecee received from {username='slave'} at 127.24.226.134:42809
I20240502 21:26:47.008793 31817 tablet_copy_service.cc:433] P 08d7a0680d244b55b93d17f9ab57b84c: ending tablet copy session 9dec645b473f40578e726f3e50e219bf-b407c2c9b3a14ce3ada52b7856c6ecee on tablet b407c2c9b3a14ce3ada52b7856c6ecee with peer 9dec645b473f40578e726f3e50e219bf
I20240502 21:26:47.011061 2872 ts_tablet_manager.cc:902] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 9dec645b473f40578e726f3e50e219bf: Initiating tablet copy from peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:47.012621 1833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e82c0ab5c441efbcfbcd383b8db82f" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:26:47.012882 1833 raft_consensus.cc:3037] T f5e82c0ab5c441efbcfbcd383b8db82f P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:47.016551 1833 raft_consensus.cc:2454] T f5e82c0ab5c441efbcfbcd383b8db82f P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 1.
I20240502 21:26:47.018649 2163 leader_election.cc:302] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:47.019026 2429 raft_consensus.cc:2781] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:47.019424 2429 raft_consensus.cc:686] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Becoming Leader. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Running, Role: LEADER
I20240502 21:26:47.019793 2429 consensus_queue.cc:237] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:47.022197 2096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e82c0ab5c441efbcfbcd383b8db82f" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:26:47.022519 2096 raft_consensus.cc:3037] T f5e82c0ab5c441efbcfbcd383b8db82f P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:47.026859 2891 leader_election.cc:288] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Requested vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:26:47.026948 31389 catalog_manager.cc:5484] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d reported cstate change: term changed from 0 to 1, leader changed from <none> to d1406e92aec84ac585304e57bd37a40d (127.24.226.136). New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:47.031914 2910 tablet_bootstrap.cc:492] T f332ab12c37f459ca85585657a0fe01e P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:47.033628 2910 tablet_bootstrap.cc:654] T f332ab12c37f459ca85585657a0fe01e P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:47.035491 2096 raft_consensus.cc:2454] T f5e82c0ab5c441efbcfbcd383b8db82f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 1.
I20240502 21:26:47.038568 2783 raft_consensus.cc:1050] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada: attempting to promote NON_VOTER 1e04157b98dd4fad98858c290b73c3db to VOTER
I20240502 21:26:47.039820 2783 consensus_queue.cc:237] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } }
I20240502 21:26:47.050173 2944 ts_tablet_manager.cc:902] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 1e04157b98dd4fad98858c290b73c3db: Initiating tablet copy from peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:47.050144 1965 raft_consensus.cc:1203] T b407c2c9b3a14ce3ada52b7856c6ecee P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:26:47.051492 31556 tablet_copy_service.cc:140] P a44f41b0b6644fb6aa29476aed8c964f: Received BeginTabletCopySession request for tablet fe4e41225da24d7c91c6bf7733a5e067 from peer 1ade0354c44c4c37ae480c7a94b11e1b ({username='slave'} at 127.24.226.135:56009)
W20240502 21:26:47.051719 31556 tablet_copy_service.cc:478] P a44f41b0b6644fb6aa29476aed8c964f: Error handling TabletCopyService RPC request from {username='slave'} at 127.24.226.135:56009: Unable to find specified tablet: fe4e41225da24d7c91c6bf7733a5e067: Not found: Tablet not found: fe4e41225da24d7c91c6bf7733a5e067
I20240502 21:26:47.053059 2506 raft_consensus.cc:482] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:47.053268 2506 raft_consensus.cc:504] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:47.054209 2506 leader_election.cc:288] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:47.055006 2230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46279270b2334633906af57c4b43e969" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:26:47.055258 2230 raft_consensus.cc:2454] T 46279270b2334633906af57c4b43e969 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 0.
I20240502 21:26:47.056526 2032 leader_election.cc:302] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:47.057909 2506 raft_consensus.cc:2781] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:47.058022 2506 raft_consensus.cc:482] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:47.058159 2506 raft_consensus.cc:3037] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:47.061672 2506 raft_consensus.cc:504] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:47.063457 2230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46279270b2334633906af57c4b43e969" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:26:47.063686 2230 raft_consensus.cc:3037] T 46279270b2334633906af57c4b43e969 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:47.066852 2230 raft_consensus.cc:2454] T 46279270b2334633906af57c4b43e969 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 1.
I20240502 21:26:47.066990 31924 consensus_queue.cc:237] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } }
I20240502 21:26:47.067401 2032 leader_election.cc:302] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:47.072527 2506 leader_election.cc:288] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 election: Requested vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:47.072754 2506 raft_consensus.cc:2781] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:47.072885 2910 tablet_bootstrap.cc:492] T f332ab12c37f459ca85585657a0fe01e P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:26:47.072952 2506 raft_consensus.cc:686] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEADER]: Becoming Leader. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Running, Role: LEADER
I20240502 21:26:47.073113 2910 ts_tablet_manager.cc:1367] T f332ab12c37f459ca85585657a0fe01e P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.000s
I20240502 21:26:47.073257 2506 consensus_queue.cc:237] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:47.074115 2910 raft_consensus.cc:348] T f332ab12c37f459ca85585657a0fe01e P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:47.075826 2910 raft_consensus.cc:374] T f332ab12c37f459ca85585657a0fe01e P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:47.075930 2910 raft_consensus.cc:729] T f332ab12c37f459ca85585657a0fe01e P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:26:47.076203 2910 consensus_queue.cc:260] T f332ab12c37f459ca85585657a0fe01e P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:47.076946 2910 ts_tablet_manager.cc:1398] T f332ab12c37f459ca85585657a0fe01e P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:26:47.083720 31535 raft_consensus.cc:1261] T 83904952a5a541ecac508449e23e70dd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:47.084532 1884 consensus_queue.cc:1035] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:26:47.087024 31862 consensus_peers.cc:485] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 -> Peer 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): Couldn't send request to peer 1ade0354c44c4c37ae480c7a94b11e1b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 83904952a5a541ecac508449e23e70dd. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:47.087028 31795 raft_consensus.cc:1261] T 83904952a5a541ecac508449e23e70dd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:47.087610 1884 consensus_queue.cc:1035] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:47.087970 1965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46279270b2334633906af57c4b43e969" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:26:47.088207 1965 raft_consensus.cc:2454] T 46279270b2334633906af57c4b43e969 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 0.
I20240502 21:26:47.088951 1965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46279270b2334633906af57c4b43e969" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf"
I20240502 21:26:47.089154 1965 raft_consensus.cc:3037] T 46279270b2334633906af57c4b43e969 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:47.092363 1965 raft_consensus.cc:2454] T 46279270b2334633906af57c4b43e969 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 1.
I20240502 21:26:47.107983 1884 raft_consensus.cc:2932] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } } }
I20240502 21:26:47.112015 31797 raft_consensus.cc:2932] T 83904952a5a541ecac508449e23e70dd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } } }
I20240502 21:26:47.118167 31389 catalog_manager.cc:5484] T 83904952a5a541ecac508449e23e70dd P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } } }
I20240502 21:26:47.135620 2891 raft_consensus.cc:482] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:47.135833 2891 raft_consensus.cc:504] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:47.137256 2096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f332ab12c37f459ca85585657a0fe01e" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
W20240502 21:26:47.137909 2163 leader_election.cc:341] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:47.138262 2891 leader_election.cc:288] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:47.138845 1965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f332ab12c37f459ca85585657a0fe01e" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:26:47.139248 1965 raft_consensus.cc:2454] T f332ab12c37f459ca85585657a0fe01e P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:26:47.139726 2161 leader_election.cc:302] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9dec645b473f40578e726f3e50e219bf, d1406e92aec84ac585304e57bd37a40d; no voters: 1ade0354c44c4c37ae480c7a94b11e1b
I20240502 21:26:47.140338 2891 raft_consensus.cc:2781] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:47.140512 2891 raft_consensus.cc:482] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:47.140657 2891 raft_consensus.cc:3037] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:47.144258 2891 raft_consensus.cc:504] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:47.145366 1965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f332ab12c37f459ca85585657a0fe01e" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf"
I20240502 21:26:47.145643 1965 raft_consensus.cc:3037] T f332ab12c37f459ca85585657a0fe01e P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:47.148916 1965 raft_consensus.cc:2454] T f332ab12c37f459ca85585657a0fe01e P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 1.
I20240502 21:26:47.149359 2161 leader_election.cc:302] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dec645b473f40578e726f3e50e219bf, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:47.149778 2429 raft_consensus.cc:2781] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:47.149936 2429 raft_consensus.cc:686] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Becoming Leader. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Running, Role: LEADER
I20240502 21:26:47.150211 2429 consensus_queue.cc:237] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:47.155015 2096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f332ab12c37f459ca85585657a0fe01e" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
W20240502 21:26:47.155612 2163 leader_election.cc:341] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:47.156004 2891 leader_election.cc:288] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Requested vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:47.177189 2938 ts_tablet_manager.cc:902] T 815b1fc784714c8cb4ba76707c345625 P 2d50f33557944bb480ea32199d9ee4f1: Initiating tablet copy from peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:47.191001 31391 catalog_manager.cc:5484] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135). New cstate: current_term: 1 leader_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:47.194434 1833 raft_consensus.cc:1261] T 182b3f15c06c4c07ade0a8854d3ecafd P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:47.195225 2608 consensus_queue.cc:1035] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:47.198577 31535 raft_consensus.cc:2932] T 83904952a5a541ecac508449e23e70dd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } } }
I20240502 21:26:47.209256 31795 raft_consensus.cc:1261] T 182b3f15c06c4c07ade0a8854d3ecafd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:47.210059 2608 consensus_queue.cc:1035] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:47.210417 31535 raft_consensus.cc:1261] T 182b3f15c06c4c07ade0a8854d3ecafd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:47.210955 2608 consensus_queue.cc:1035] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:47.217335 31391 catalog_manager.cc:5484] T 4f09394db52e489e9bc42a4883ee8b1c P 1e04157b98dd4fad98858c290b73c3db reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e04157b98dd4fad98858c290b73c3db (127.24.226.133). New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:47.223306 2957 ts_tablet_manager.cc:902] T 5b1dcae186f542cea92c9191fa8ef951 P d1406e92aec84ac585304e57bd37a40d: Initiating tablet copy from peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:47.235944 2608 raft_consensus.cc:2932] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:47.238499 31797 raft_consensus.cc:2932] T 182b3f15c06c4c07ade0a8854d3ecafd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:47.245934 31391 catalog_manager.cc:5484] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d reported cstate change: term changed from 0 to 1, leader changed from <none> to d1406e92aec84ac585304e57bd37a40d (127.24.226.136). New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:47.251634 31389 catalog_manager.cc:5484] T 182b3f15c06c4c07ade0a8854d3ecafd P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index 4 to 5, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:47.265105 2777 raft_consensus.cc:482] T 4f09394db52e489e9bc42a4883ee8b1c P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:47.265267 2777 raft_consensus.cc:504] T 4f09394db52e489e9bc42a4883ee8b1c P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:47.266021 2777 leader_election.cc:288] T 4f09394db52e489e9bc42a4883ee8b1c P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:47.267311 1833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f09394db52e489e9bc42a4883ee8b1c" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:26:47.268025 2096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f09394db52e489e9bc42a4883ee8b1c" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:47.268350 2096 raft_consensus.cc:2379] T 4f09394db52e489e9bc42a4883ee8b1c P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2d50f33557944bb480ea32199d9ee4f1 in current term 1: Already voted for candidate 1e04157b98dd4fad98858c290b73c3db in this term.
I20240502 21:26:47.269044 31862 leader_election.cc:302] T 4f09394db52e489e9bc42a4883ee8b1c P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1; no voters: 1ade0354c44c4c37ae480c7a94b11e1b, 1e04157b98dd4fad98858c290b73c3db
I20240502 21:26:47.269584 2777 raft_consensus.cc:3037] T 4f09394db52e489e9bc42a4883ee8b1c P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:47.269811 2898 tablet_bootstrap.cc:492] T f332ab12c37f459ca85585657a0fe01e P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:47.270058 2898 ts_tablet_manager.cc:1367] T f332ab12c37f459ca85585657a0fe01e P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.320s user 0.001s sys 0.008s
I20240502 21:26:47.270990 2898 raft_consensus.cc:348] T f332ab12c37f459ca85585657a0fe01e P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:47.271301 2898 raft_consensus.cc:374] T f332ab12c37f459ca85585657a0fe01e P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:47.271689 2898 raft_consensus.cc:729] T f332ab12c37f459ca85585657a0fe01e P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:47.272038 2898 consensus_queue.cc:260] T f332ab12c37f459ca85585657a0fe01e P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:47.272290 2974 raft_consensus.cc:1050] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada: attempting to promote NON_VOTER 2d50f33557944bb480ea32199d9ee4f1 to VOTER
I20240502 21:26:47.273145 2974 consensus_queue.cc:237] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } }
I20240502 21:26:47.273443 2777 raft_consensus.cc:2726] T 4f09394db52e489e9bc42a4883ee8b1c P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:26:47.275554 2967 ts_tablet_manager.cc:902] T 62f083e9641042c990efcfb1c4104de5 P d1406e92aec84ac585304e57bd37a40d: Initiating tablet copy from peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:47.276506 2898 ts_tablet_manager.cc:1398] T f332ab12c37f459ca85585657a0fe01e P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 21:26:47.277101 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f09394db52e489e9bc42a4883ee8b1c" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:47.277416 31926 raft_consensus.cc:2454] T 4f09394db52e489e9bc42a4883ee8b1c P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:26:47.279407 31926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f09394db52e489e9bc42a4883ee8b1c" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:47.282776 31926 raft_consensus.cc:2454] T 4f09394db52e489e9bc42a4883ee8b1c P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:26:47.284147 1833 raft_consensus.cc:2932] T 182b3f15c06c4c07ade0a8854d3ecafd P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:47.285320 2957 tablet_copy_client.cc:308] T 5b1dcae186f542cea92c9191fa8ef951 P d1406e92aec84ac585304e57bd37a40d: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.131:40807
I20240502 21:26:47.285697 2230 raft_consensus.cc:1261] T 49acc5301b9849e9b6795179acbc59a8 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:47.287535 2976 consensus_queue.cc:1035] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:47.287725 2096 raft_consensus.cc:1261] T 49acc5301b9849e9b6795179acbc59a8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:47.287534 31535 raft_consensus.cc:2932] T 182b3f15c06c4c07ade0a8854d3ecafd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:47.292490 2608 consensus_queue.cc:1035] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:47.293359 31390 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240502 21:26:47.293572 31390 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240502 21:26:47.293701 31666 consensus_queue.cc:237] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } }
I20240502 21:26:47.296836 1834 raft_consensus.cc:1261] T 182b3f15c06c4c07ade0a8854d3ecafd P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:47.298784 31390 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet a4575a0989ac4140aec6b2cdb1e92032
I20240502 21:26:47.299397 31390 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 2ccfe8f0b1dd450db470891faebbff99
I20240502 21:26:47.300742 31390 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet c0b0e63252b34943984f1898e725e602
I20240502 21:26:47.300846 31775 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ccfe8f0b1dd450db470891faebbff99 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:47 UTC) from {username='slave'} at 127.0.0.1:51596
I20240502 21:26:47.301229 2907 tablet_replica.cc:330] T 2ccfe8f0b1dd450db470891faebbff99 P 08d7a0680d244b55b93d17f9ab57b84c: stopping tablet replica
I20240502 21:26:47.302039 31905 tablet_service.cc:1514] Processing DeleteTablet for tablet a4575a0989ac4140aec6b2cdb1e92032 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:47 UTC) from {username='slave'} at 127.0.0.1:47824
I20240502 21:26:47.302269 31516 tablet_service.cc:1514] Processing DeleteTablet for tablet a4575a0989ac4140aec6b2cdb1e92032 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:47 UTC) from {username='slave'} at 127.0.0.1:45194
I20240502 21:26:47.302373 31906 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ccfe8f0b1dd450db470891faebbff99 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:47 UTC) from {username='slave'} at 127.0.0.1:47824
I20240502 21:26:47.302484 31645 tablet_service.cc:1514] Processing DeleteTablet for tablet a4575a0989ac4140aec6b2cdb1e92032 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:47 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:47.302701 2328 tablet_replica.cc:330] T a4575a0989ac4140aec6b2cdb1e92032 P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:26:47.302743 2602 tablet_replica.cc:330] T a4575a0989ac4140aec6b2cdb1e92032 P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:47.302767 31647 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ccfe8f0b1dd450db470891faebbff99 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:47 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:47.302956 2936 ts_tablet_manager.cc:902] T f9bbc56ba2f841b9bb36a864e1a1f808 P 9dec645b473f40578e726f3e50e219bf: Initiating tablet copy from peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:47.302981 2328 raft_consensus.cc:2229] T a4575a0989ac4140aec6b2cdb1e92032 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:47.303018 2602 raft_consensus.cc:2229] T a4575a0989ac4140aec6b2cdb1e92032 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:47.303241 2602 raft_consensus.cc:2258] T a4575a0989ac4140aec6b2cdb1e92032 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:47.303503 2328 raft_consensus.cc:2258] T a4575a0989ac4140aec6b2cdb1e92032 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:47.304162 2602 ts_tablet_manager.cc:1875] T a4575a0989ac4140aec6b2cdb1e92032 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:47.304211 31647 tablet_service.cc:1514] Processing DeleteTablet for tablet c0b0e63252b34943984f1898e725e602 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:47 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:47.304322 2328 ts_tablet_manager.cc:1875] T a4575a0989ac4140aec6b2cdb1e92032 P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:47.304690 2329 tablet_replica.cc:330] T a4575a0989ac4140aec6b2cdb1e92032 P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:47.304936 2329 raft_consensus.cc:2229] T a4575a0989ac4140aec6b2cdb1e92032 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:47.305225 2329 raft_consensus.cc:2258] T a4575a0989ac4140aec6b2cdb1e92032 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:47.305948 2329 ts_tablet_manager.cc:1875] T a4575a0989ac4140aec6b2cdb1e92032 P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 21:26:47.305938 31862 consensus_peers.cc:485] T 2ccfe8f0b1dd450db470891faebbff99 P 2d50f33557944bb480ea32199d9ee4f1 -> Peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): Couldn't send request to peer 08d7a0680d244b55b93d17f9ab57b84c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:47.308645 31516 tablet_service.cc:1514] Processing DeleteTablet for tablet c0b0e63252b34943984f1898e725e602 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:47 UTC) from {username='slave'} at 127.0.0.1:45194
I20240502 21:26:47.309557 31906 tablet_service.cc:1514] Processing DeleteTablet for tablet c0b0e63252b34943984f1898e725e602 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:47 UTC) from {username='slave'} at 127.0.0.1:47824
I20240502 21:26:47.310460 31534 raft_consensus.cc:1261] T 182b3f15c06c4c07ade0a8854d3ecafd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:47.311547 2907 raft_consensus.cc:2229] T 2ccfe8f0b1dd450db470891faebbff99 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:47.311851 2907 raft_consensus.cc:2258] T 2ccfe8f0b1dd450db470891faebbff99 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:47.311846 2976 consensus_queue.cc:1035] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:47.312451 2974 consensus_queue.cc:1035] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:47.312925 2907 ts_tablet_manager.cc:1875] T 2ccfe8f0b1dd450db470891faebbff99 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:47.313431 2602 ts_tablet_manager.cc:1888] T a4575a0989ac4140aec6b2cdb1e92032 P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:47.313570 2602 log.cc:1196] T a4575a0989ac4140aec6b2cdb1e92032 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/a4575a0989ac4140aec6b2cdb1e92032
I20240502 21:26:47.314002 2602 ts_tablet_manager.cc:1909] T a4575a0989ac4140aec6b2cdb1e92032 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:26:47.314327 2783 raft_consensus.cc:2932] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:47.315055 1833 raft_consensus.cc:2932] T 182b3f15c06c4c07ade0a8854d3ecafd P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:47.318324 31382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 182b3f15c06c4c07ade0a8854d3ecafd with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:47.318080 2938 tablet_copy_client.cc:308] T 815b1fc784714c8cb4ba76707c345625 P 2d50f33557944bb480ea32199d9ee4f1: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.131:40807
I20240502 21:26:47.319555 31817 tablet_copy_service.cc:140] P 08d7a0680d244b55b93d17f9ab57b84c: Received BeginTabletCopySession request for tablet 815b1fc784714c8cb4ba76707c345625 from peer 2d50f33557944bb480ea32199d9ee4f1 ({username='slave'} at 127.24.226.132:44555)
I20240502 21:26:47.319737 31817 tablet_copy_service.cc:161] P 08d7a0680d244b55b93d17f9ab57b84c: Beginning new tablet copy session on tablet 815b1fc784714c8cb4ba76707c345625 from peer 2d50f33557944bb480ea32199d9ee4f1 at {username='slave'} at 127.24.226.132:44555: session id = 2d50f33557944bb480ea32199d9ee4f1-815b1fc784714c8cb4ba76707c345625
I20240502 21:26:47.322609 31817 tablet_copy_source_session.cc:205] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:47.323661 31534 raft_consensus.cc:2932] T 182b3f15c06c4c07ade0a8854d3ecafd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:47.327507 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet a4575a0989ac4140aec6b2cdb1e92032 (table rebalance_test_table_18 [id=1fffea26d24e40e88ee6944f57c34b08]) successfully deleted
I20240502 21:26:47.327935 31389 catalog_manager.cc:5484] T 182b3f15c06c4c07ade0a8854d3ecafd P 1e04157b98dd4fad98858c290b73c3db reported cstate change: config changed from index 5 to 6, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:47.328946 2602 tablet_replica.cc:330] T 2ccfe8f0b1dd450db470891faebbff99 P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:47.329151 2602 raft_consensus.cc:2229] T 2ccfe8f0b1dd450db470891faebbff99 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:47.329411 2602 raft_consensus.cc:2258] T 2ccfe8f0b1dd450db470891faebbff99 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:47.330066 2602 ts_tablet_manager.cc:1875] T 2ccfe8f0b1dd450db470891faebbff99 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:47.330536 2938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 815b1fc784714c8cb4ba76707c345625. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:47.335577 2907 ts_tablet_manager.cc:1888] T 2ccfe8f0b1dd450db470891faebbff99 P 08d7a0680d244b55b93d17f9ab57b84c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:47.335988 31775 tablet_service.cc:1514] Processing DeleteTablet for tablet 182b3f15c06c4c07ade0a8854d3ecafd with delete_type TABLET_DATA_TOMBSTONED (TS 08d7a0680d244b55b93d17f9ab57b84c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51596
I20240502 21:26:47.336321 2907 log.cc:1196] T 2ccfe8f0b1dd450db470891faebbff99 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/wals/2ccfe8f0b1dd450db470891faebbff99
I20240502 21:26:47.337306 2907 ts_tablet_manager.cc:1909] T 2ccfe8f0b1dd450db470891faebbff99 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting consensus metadata
I20240502 21:26:47.340917 31924 raft_consensus.cc:1261] T 49acc5301b9849e9b6795179acbc59a8 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEARNER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:47.341470 2975 consensus_queue.cc:1035] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 21:26:47.352453 2329 ts_tablet_manager.cc:1888] T a4575a0989ac4140aec6b2cdb1e92032 P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:47.352725 2329 log.cc:1196] T a4575a0989ac4140aec6b2cdb1e92032 P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/a4575a0989ac4140aec6b2cdb1e92032
I20240502 21:26:47.353332 2329 ts_tablet_manager.cc:1909] T a4575a0989ac4140aec6b2cdb1e92032 P a44f41b0b6644fb6aa29476aed8c964f: Deleting consensus metadata
I20240502 21:26:47.354537 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet a4575a0989ac4140aec6b2cdb1e92032 (table rebalance_test_table_18 [id=1fffea26d24e40e88ee6944f57c34b08]) successfully deleted
I20240502 21:26:47.356981 2944 tablet_copy_client.cc:308] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.131:40807
I20240502 21:26:47.357831 31817 tablet_copy_service.cc:140] P 08d7a0680d244b55b93d17f9ab57b84c: Received BeginTabletCopySession request for tablet 9ea8ddb9f8e04893b56c10dbbb5af321 from peer 1e04157b98dd4fad98858c290b73c3db ({username='slave'} at 127.24.226.133:41593)
I20240502 21:26:47.358004 31817 tablet_copy_service.cc:161] P 08d7a0680d244b55b93d17f9ab57b84c: Beginning new tablet copy session on tablet 9ea8ddb9f8e04893b56c10dbbb5af321 from peer 1e04157b98dd4fad98858c290b73c3db at {username='slave'} at 127.24.226.133:41593: session id = 1e04157b98dd4fad98858c290b73c3db-9ea8ddb9f8e04893b56c10dbbb5af321
I20240502 21:26:47.360100 2938 tablet_copy_client.cc:783] T 815b1fc784714c8cb4ba76707c345625 P 2d50f33557944bb480ea32199d9ee4f1: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:47.360370 2938 tablet_copy_client.cc:647] T 815b1fc784714c8cb4ba76707c345625 P 2d50f33557944bb480ea32199d9ee4f1: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:47.363026 31817 tablet_copy_source_session.cc:205] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:47.364640 2938 tablet_copy_client.cc:523] T 815b1fc784714c8cb4ba76707c345625 P 2d50f33557944bb480ea32199d9ee4f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:47.364820 2506 raft_consensus.cc:482] T f332ab12c37f459ca85585657a0fe01e P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:47.364982 2506 raft_consensus.cc:504] T f332ab12c37f459ca85585657a0fe01e P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:47.365329 2944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ea8ddb9f8e04893b56c10dbbb5af321. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:47.365701 2506 leader_election.cc:288] T f332ab12c37f459ca85585657a0fe01e P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:26:47.366142 2328 ts_tablet_manager.cc:1888] T a4575a0989ac4140aec6b2cdb1e92032 P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:47.366341 2328 log.cc:1196] T a4575a0989ac4140aec6b2cdb1e92032 P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/a4575a0989ac4140aec6b2cdb1e92032
I20240502 21:26:47.366678 2228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f332ab12c37f459ca85585657a0fe01e" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:26:47.366968 2328 ts_tablet_manager.cc:1909] T a4575a0989ac4140aec6b2cdb1e92032 P 2d50f33557944bb480ea32199d9ee4f1: Deleting consensus metadata
I20240502 21:26:47.367208 1966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f332ab12c37f459ca85585657a0fe01e" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:26:47.367509 1966 raft_consensus.cc:2379] T f332ab12c37f459ca85585657a0fe01e P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ade0354c44c4c37ae480c7a94b11e1b in current term 1: Already voted for candidate d1406e92aec84ac585304e57bd37a40d in this term.
I20240502 21:26:47.368274 2030 leader_election.cc:302] T f332ab12c37f459ca85585657a0fe01e P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b; no voters: 9dec645b473f40578e726f3e50e219bf, d1406e92aec84ac585304e57bd37a40d
I20240502 21:26:47.368428 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet a4575a0989ac4140aec6b2cdb1e92032 (table rebalance_test_table_18 [id=1fffea26d24e40e88ee6944f57c34b08]) successfully deleted
I20240502 21:26:47.368713 2506 raft_consensus.cc:3037] T f332ab12c37f459ca85585657a0fe01e P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:47.369279 2328 tablet_replica.cc:330] T 2ccfe8f0b1dd450db470891faebbff99 P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:26:47.369647 2328 raft_consensus.cc:2229] T 2ccfe8f0b1dd450db470891faebbff99 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:47.370121 2328 raft_consensus.cc:2258] T 2ccfe8f0b1dd450db470891faebbff99 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:47.370878 2328 ts_tablet_manager.cc:1875] T 2ccfe8f0b1dd450db470891faebbff99 P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:47.371351 2506 raft_consensus.cc:2726] T f332ab12c37f459ca85585657a0fe01e P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:26:47.371920 2975 raft_consensus.cc:2932] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:26:47.373829 31381 catalog_manager.cc:4830] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): tablet 2ccfe8f0b1dd450db470891faebbff99 (table rebalance_test_table_18 [id=1fffea26d24e40e88ee6944f57c34b08]) successfully deleted
I20240502 21:26:47.373930 2907 tablet_replica.cc:330] T 182b3f15c06c4c07ade0a8854d3ecafd P 08d7a0680d244b55b93d17f9ab57b84c: stopping tablet replica
I20240502 21:26:47.374159 2907 raft_consensus.cc:2229] T 182b3f15c06c4c07ade0a8854d3ecafd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:47.374457 2907 raft_consensus.cc:2258] T 182b3f15c06c4c07ade0a8854d3ecafd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:47.375465 2907 ts_tablet_manager.cc:1875] T 182b3f15c06c4c07ade0a8854d3ecafd P 08d7a0680d244b55b93d17f9ab57b84c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:47.379860 2329 tablet_replica.cc:330] T c0b0e63252b34943984f1898e725e602 P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:47.380190 2329 raft_consensus.cc:2229] T c0b0e63252b34943984f1898e725e602 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:47.380538 2329 raft_consensus.cc:2258] T c0b0e63252b34943984f1898e725e602 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:47.381467 2329 ts_tablet_manager.cc:1875] T c0b0e63252b34943984f1898e725e602 P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:47.381592 2096 raft_consensus.cc:2932] T 49acc5301b9849e9b6795179acbc59a8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:26:47.382364 31389 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:26:47.385286 2602 ts_tablet_manager.cc:1888] T 2ccfe8f0b1dd450db470891faebbff99 P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:47.385406 2602 log.cc:1196] T 2ccfe8f0b1dd450db470891faebbff99 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/2ccfe8f0b1dd450db470891faebbff99
I20240502 21:26:47.385845 2602 ts_tablet_manager.cc:1909] T 2ccfe8f0b1dd450db470891faebbff99 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:26:47.386816 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet 2ccfe8f0b1dd450db470891faebbff99 (table rebalance_test_table_18 [id=1fffea26d24e40e88ee6944f57c34b08]) successfully deleted
I20240502 21:26:47.389199 2931 tablet_copy_client.cc:308] T b9342dbe0497499ba4c238f2ed033293 P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.133:39863
I20240502 21:26:47.389775 31391 catalog_manager.cc:5484] T 49acc5301b9849e9b6795179acbc59a8 P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: config changed from index 2 to 3, 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:26:47.390453 2230 raft_consensus.cc:2932] T 49acc5301b9849e9b6795179acbc59a8 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:26:47.391355 1854 tablet_copy_service.cc:140] P 1e04157b98dd4fad98858c290b73c3db: Received BeginTabletCopySession request for tablet b9342dbe0497499ba4c238f2ed033293 from peer 05f4fac41c4f4959b9b316cb112d0ada ({username='slave'} at 127.24.226.130:43577)
I20240502 21:26:47.391618 1854 tablet_copy_service.cc:161] P 1e04157b98dd4fad98858c290b73c3db: Beginning new tablet copy session on tablet b9342dbe0497499ba4c238f2ed033293 from peer 05f4fac41c4f4959b9b316cb112d0ada at {username='slave'} at 127.24.226.130:43577: session id = 05f4fac41c4f4959b9b316cb112d0ada-b9342dbe0497499ba4c238f2ed033293
I20240502 21:26:47.393321 2095 raft_consensus.cc:1261] T f332ab12c37f459ca85585657a0fe01e P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:47.393358 1966 raft_consensus.cc:1261] T f332ab12c37f459ca85585657a0fe01e P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:47.394047 2429 consensus_queue.cc:1035] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:47.394498 2429 consensus_queue.cc:1035] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:47.381246 2938 tablet_bootstrap.cc:492] T 815b1fc784714c8cb4ba76707c345625 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:47.399286 1854 tablet_copy_source_session.cc:205] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:47.403620 2602 tablet_replica.cc:330] T c0b0e63252b34943984f1898e725e602 P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:47.404106 2602 raft_consensus.cc:2229] T c0b0e63252b34943984f1898e725e602 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:47.406641 2602 raft_consensus.cc:2258] T c0b0e63252b34943984f1898e725e602 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:47.407189 31666 consensus_queue.cc:237] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } }
I20240502 21:26:47.410254 31817 tablet_copy_service.cc:140] P 08d7a0680d244b55b93d17f9ab57b84c: Received BeginTabletCopySession request for tablet 5b1dcae186f542cea92c9191fa8ef951 from peer d1406e92aec84ac585304e57bd37a40d ({username='slave'} at 127.24.226.136:57795)
I20240502 21:26:47.410437 31817 tablet_copy_service.cc:161] P 08d7a0680d244b55b93d17f9ab57b84c: Beginning new tablet copy session on tablet 5b1dcae186f542cea92c9191fa8ef951 from peer d1406e92aec84ac585304e57bd37a40d at {username='slave'} at 127.24.226.136:57795: session id = d1406e92aec84ac585304e57bd37a40d-5b1dcae186f542cea92c9191fa8ef951
I20240502 21:26:47.412619 2602 ts_tablet_manager.cc:1875] T c0b0e63252b34943984f1898e725e602 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:47.413530 31817 tablet_copy_source_session.cc:205] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:47.415632 2957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b1dcae186f542cea92c9191fa8ef951. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:47.418324 2967 tablet_copy_client.cc:308] T 62f083e9641042c990efcfb1c4104de5 P d1406e92aec84ac585304e57bd37a40d: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.132:35933
I20240502 21:26:47.422935 2957 tablet_copy_client.cc:783] T 5b1dcae186f542cea92c9191fa8ef951 P d1406e92aec84ac585304e57bd37a40d: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:47.423610 2957 tablet_copy_client.cc:647] T 5b1dcae186f542cea92c9191fa8ef951 P d1406e92aec84ac585304e57bd37a40d: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:47.424783 2931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9342dbe0497499ba4c238f2ed033293. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:47.427574 2602 ts_tablet_manager.cc:1888] T c0b0e63252b34943984f1898e725e602 P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:47.427732 2602 log.cc:1196] T c0b0e63252b34943984f1898e725e602 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/c0b0e63252b34943984f1898e725e602
I20240502 21:26:47.428164 2602 ts_tablet_manager.cc:1909] T c0b0e63252b34943984f1898e725e602 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:26:47.429546 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet c0b0e63252b34943984f1898e725e602 (table rebalance_test_table_18 [id=1fffea26d24e40e88ee6944f57c34b08]) successfully deleted
I20240502 21:26:47.431986 2230 raft_consensus.cc:1261] T 49acc5301b9849e9b6795179acbc59a8 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:47.432600 2974 consensus_queue.cc:1035] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:47.432297 31924 raft_consensus.cc:2932] T 49acc5301b9849e9b6795179acbc59a8 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:26:47.435796 2872 tablet_copy_client.cc:308] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 9dec645b473f40578e726f3e50e219bf: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.129:41039
I20240502 21:26:47.436879 31556 tablet_copy_service.cc:140] P a44f41b0b6644fb6aa29476aed8c964f: Received BeginTabletCopySession request for tablet 55fc18e4cc98446ca50c1dd2b6a9a89d from peer 9dec645b473f40578e726f3e50e219bf ({username='slave'} at 127.24.226.134:40609)
I20240502 21:26:47.437057 31556 tablet_copy_service.cc:161] P a44f41b0b6644fb6aa29476aed8c964f: Beginning new tablet copy session on tablet 55fc18e4cc98446ca50c1dd2b6a9a89d from peer 9dec645b473f40578e726f3e50e219bf at {username='slave'} at 127.24.226.134:40609: session id = 9dec645b473f40578e726f3e50e219bf-55fc18e4cc98446ca50c1dd2b6a9a89d
I20240502 21:26:47.441995 2968 raft_consensus.cc:2932] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:26:47.442569 31556 tablet_copy_source_session.cc:205] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:47.444013 31926 raft_consensus.cc:1261] T 49acc5301b9849e9b6795179acbc59a8 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:47.444147 2872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55fc18e4cc98446ca50c1dd2b6a9a89d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:47.445124 2230 raft_consensus.cc:2932] T 49acc5301b9849e9b6795179acbc59a8 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:26:47.445999 31382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 49acc5301b9849e9b6795179acbc59a8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:47.448450 31389 catalog_manager.cc:5484] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:47.449929 2968 consensus_queue.cc:1035] T 49acc5301b9849e9b6795179acbc59a8 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:47.452425 2075 tablet_service.cc:1514] Processing DeleteTablet for tablet 49acc5301b9849e9b6795179acbc59a8 with delete_type TABLET_DATA_TOMBSTONED (TS 1ade0354c44c4c37ae480c7a94b11e1b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36740
I20240502 21:26:47.452491 31926 raft_consensus.cc:2932] T 49acc5301b9849e9b6795179acbc59a8 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:26:47.454406 2957 tablet_copy_client.cc:523] T 5b1dcae186f542cea92c9191fa8ef951 P d1406e92aec84ac585304e57bd37a40d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:47.459681 2957 tablet_bootstrap.cc:492] T 5b1dcae186f542cea92c9191fa8ef951 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:47.459877 2872 tablet_copy_client.cc:783] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 9dec645b473f40578e726f3e50e219bf: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:47.460095 2872 tablet_copy_client.cc:647] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 9dec645b473f40578e726f3e50e219bf: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:47.466802 2872 tablet_copy_client.cc:523] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 9dec645b473f40578e726f3e50e219bf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:47.466955 2907 ts_tablet_manager.cc:1888] T 182b3f15c06c4c07ade0a8854d3ecafd P 08d7a0680d244b55b93d17f9ab57b84c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:26:47.467135 2907 log.cc:1196] T 182b3f15c06c4c07ade0a8854d3ecafd P 08d7a0680d244b55b93d17f9ab57b84c: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/wals/182b3f15c06c4c07ade0a8854d3ecafd
I20240502 21:26:47.470602 2872 tablet_bootstrap.cc:492] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:47.476425 2329 ts_tablet_manager.cc:1888] T c0b0e63252b34943984f1898e725e602 P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:47.476867 31381 catalog_manager.cc:4830] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): tablet 182b3f15c06c4c07ade0a8854d3ecafd (table rebalance_test_table_3 [id=59305f5894b34d0085a6cada5a676ca4]) successfully deleted
I20240502 21:26:47.477880 2329 log.cc:1196] T c0b0e63252b34943984f1898e725e602 P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/c0b0e63252b34943984f1898e725e602
I20240502 21:26:47.479061 2329 ts_tablet_manager.cc:1909] T c0b0e63252b34943984f1898e725e602 P a44f41b0b6644fb6aa29476aed8c964f: Deleting consensus metadata
I20240502 21:26:47.479673 3007 tablet_replica.cc:330] T 49acc5301b9849e9b6795179acbc59a8 P 1ade0354c44c4c37ae480c7a94b11e1b: stopping tablet replica
I20240502 21:26:47.479944 3007 raft_consensus.cc:2229] T 49acc5301b9849e9b6795179acbc59a8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:47.480288 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet c0b0e63252b34943984f1898e725e602 (table rebalance_test_table_18 [id=1fffea26d24e40e88ee6944f57c34b08]) successfully deleted
I20240502 21:26:47.480746 3007 raft_consensus.cc:2258] T 49acc5301b9849e9b6795179acbc59a8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:47.482758 31389 catalog_manager.cc:5576] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753) does not have the latest schema for tablet f332ab12c37f459ca85585657a0fe01e (table rebalancer_dynamic_table [id=b6e29d3838874defa6668514ac1c2423]). Expected version 1 got 0
I20240502 21:26:47.482960 31947 tablet_copy_service.cc:140] P 2d50f33557944bb480ea32199d9ee4f1: Received BeginTabletCopySession request for tablet 62f083e9641042c990efcfb1c4104de5 from peer d1406e92aec84ac585304e57bd37a40d ({username='slave'} at 127.24.226.136:43115)
I20240502 21:26:47.483167 31947 tablet_copy_service.cc:161] P 2d50f33557944bb480ea32199d9ee4f1: Beginning new tablet copy session on tablet 62f083e9641042c990efcfb1c4104de5 from peer d1406e92aec84ac585304e57bd37a40d at {username='slave'} at 127.24.226.136:43115: session id = d1406e92aec84ac585304e57bd37a40d-62f083e9641042c990efcfb1c4104de5
I20240502 21:26:47.486410 31947 tablet_copy_source_session.cc:205] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:47.487890 2967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62f083e9641042c990efcfb1c4104de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:47.497740 2967 tablet_copy_client.cc:783] T 62f083e9641042c990efcfb1c4104de5 P d1406e92aec84ac585304e57bd37a40d: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:47.498034 2967 tablet_copy_client.cc:647] T 62f083e9641042c990efcfb1c4104de5 P d1406e92aec84ac585304e57bd37a40d: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:47.500839 2967 tablet_copy_client.cc:523] T 62f083e9641042c990efcfb1c4104de5 P d1406e92aec84ac585304e57bd37a40d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:47.506198 2967 tablet_bootstrap.cc:492] T 62f083e9641042c990efcfb1c4104de5 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:47.508890 31389 catalog_manager.cc:5576] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569) does not have the latest schema for tablet f332ab12c37f459ca85585657a0fe01e (table rebalancer_dynamic_table [id=b6e29d3838874defa6668514ac1c2423]). Expected version 1 got 0
I20240502 21:26:47.513257 2938 tablet_bootstrap.cc:492] T 815b1fc784714c8cb4ba76707c345625 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:47.513896 2938 tablet_bootstrap.cc:492] T 815b1fc784714c8cb4ba76707c345625 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap complete.
I20240502 21:26:47.514264 2938 ts_tablet_manager.cc:1367] T 815b1fc784714c8cb4ba76707c345625 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.133s user 0.006s sys 0.009s
I20240502 21:26:47.515326 2938 raft_consensus.cc:348] T 815b1fc784714c8cb4ba76707c345625 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } }
I20240502 21:26:47.516791 2938 raft_consensus.cc:729] T 815b1fc784714c8cb4ba76707c345625 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: LEARNER
I20240502 21:26:47.515925 2328 ts_tablet_manager.cc:1888] T 2ccfe8f0b1dd450db470891faebbff99 P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:47.517204 2328 log.cc:1196] T 2ccfe8f0b1dd450db470891faebbff99 P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/2ccfe8f0b1dd450db470891faebbff99
I20240502 21:26:47.517706 2328 ts_tablet_manager.cc:1909] T 2ccfe8f0b1dd450db470891faebbff99 P 2d50f33557944bb480ea32199d9ee4f1: Deleting consensus metadata
I20240502 21:26:47.517933 2938 consensus_queue.cc:260] T 815b1fc784714c8cb4ba76707c345625 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } }
I20240502 21:26:47.520320 2938 ts_tablet_manager.cc:1398] T 815b1fc784714c8cb4ba76707c345625 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 21:26:47.521204 2328 tablet_replica.cc:330] T c0b0e63252b34943984f1898e725e602 P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:26:47.521263 2999 mvcc.cc:205] Tried to move back new op lower bound from 7023350609464217600 to 7023350608489439232. Current Snapshot: MvccSnapshot[applied={T|T < 7023350609464217600}]
I20240502 21:26:47.521453 31817 tablet_copy_service.cc:342] P 08d7a0680d244b55b93d17f9ab57b84c: Request end of tablet copy session 2d50f33557944bb480ea32199d9ee4f1-815b1fc784714c8cb4ba76707c345625 received from {username='slave'} at 127.24.226.132:44555
I20240502 21:26:47.521486 2328 raft_consensus.cc:2229] T c0b0e63252b34943984f1898e725e602 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:47.521579 31817 tablet_copy_service.cc:433] P 08d7a0680d244b55b93d17f9ab57b84c: ending tablet copy session 2d50f33557944bb480ea32199d9ee4f1-815b1fc784714c8cb4ba76707c345625 on tablet 815b1fc784714c8cb4ba76707c345625 with peer 2d50f33557944bb480ea32199d9ee4f1
I20240502 21:26:47.521732 2328 raft_consensus.cc:2258] T c0b0e63252b34943984f1898e725e602 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:47.522348 2328 ts_tablet_manager.cc:1875] T c0b0e63252b34943984f1898e725e602 P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:47.523273 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet 2ccfe8f0b1dd450db470891faebbff99 (table rebalance_test_table_18 [id=1fffea26d24e40e88ee6944f57c34b08]) successfully deleted
I20240502 21:26:47.530375 3010 tablet.cc:1722] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:47.531111 3010 tablet.cc:1620] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d: MemRowSet was empty: no flush needed.
I20240502 21:26:47.536389 2944 tablet_copy_client.cc:783] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:47.536736 2944 tablet_copy_client.cc:647] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:47.543736 2931 tablet_copy_client.cc:783] T b9342dbe0497499ba4c238f2ed033293 P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:47.544142 2931 tablet_copy_client.cc:647] T b9342dbe0497499ba4c238f2ed033293 P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:47.528904 3003 mvcc.cc:205] Tried to move back new op lower bound from 7023350609464217600 to 7023350608489439232. Current Snapshot: MvccSnapshot[applied={T|T < 7023350609464217600}]
I20240502 21:26:47.546638 2328 ts_tablet_manager.cc:1888] T c0b0e63252b34943984f1898e725e602 P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 21:26:47.546777 2328 log.cc:1196] T c0b0e63252b34943984f1898e725e602 P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/c0b0e63252b34943984f1898e725e602
I20240502 21:26:47.547230 2328 ts_tablet_manager.cc:1909] T c0b0e63252b34943984f1898e725e602 P 2d50f33557944bb480ea32199d9ee4f1: Deleting consensus metadata
I20240502 21:26:47.547377 2944 tablet_copy_client.cc:523] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:47.553269 3007 ts_tablet_manager.cc:1875] T 49acc5301b9849e9b6795179acbc59a8 P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:47.553309 3016 tablet.cc:1722] T f332ab12c37f459ca85585657a0fe01e P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:47.553920 3016 tablet.cc:1620] T f332ab12c37f459ca85585657a0fe01e P 9dec645b473f40578e726f3e50e219bf: MemRowSet was empty: no flush needed.
I20240502 21:26:47.553951 2931 tablet_copy_client.cc:523] T b9342dbe0497499ba4c238f2ed033293 P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:47.555804 2944 tablet_bootstrap.cc:492] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:47.556061 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet c0b0e63252b34943984f1898e725e602 (table rebalance_test_table_18 [id=1fffea26d24e40e88ee6944f57c34b08]) successfully deleted
I20240502 21:26:47.559139 3011 tablet.cc:1722] T f332ab12c37f459ca85585657a0fe01e P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:47.559450 31389 catalog_manager.cc:5576] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569) does not have the latest schema for tablet f332ab12c37f459ca85585657a0fe01e (table rebalancer_dynamic_table [id=b6e29d3838874defa6668514ac1c2423]). Expected version 1 got 0
I20240502 21:26:47.559676 3011 tablet.cc:1620] T f332ab12c37f459ca85585657a0fe01e P 1ade0354c44c4c37ae480c7a94b11e1b: MemRowSet was empty: no flush needed.
I20240502 21:26:47.559935 31395 catalog_manager.cc:5576] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753) does not have the latest schema for tablet f332ab12c37f459ca85585657a0fe01e (table rebalancer_dynamic_table [id=b6e29d3838874defa6668514ac1c2423]). Expected version 1 got 0
I20240502 21:26:47.562754 2936 tablet_copy_client.cc:308] T f9bbc56ba2f841b9bb36a864e1a1f808 P 9dec645b473f40578e726f3e50e219bf: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.131:40807
I20240502 21:26:47.563465 31817 tablet_copy_service.cc:140] P 08d7a0680d244b55b93d17f9ab57b84c: Received BeginTabletCopySession request for tablet f9bbc56ba2f841b9bb36a864e1a1f808 from peer 9dec645b473f40578e726f3e50e219bf ({username='slave'} at 127.24.226.134:42809)
I20240502 21:26:47.563628 31817 tablet_copy_service.cc:161] P 08d7a0680d244b55b93d17f9ab57b84c: Beginning new tablet copy session on tablet f9bbc56ba2f841b9bb36a864e1a1f808 from peer 9dec645b473f40578e726f3e50e219bf at {username='slave'} at 127.24.226.134:42809: session id = 9dec645b473f40578e726f3e50e219bf-f9bbc56ba2f841b9bb36a864e1a1f808
I20240502 21:26:47.566372 31817 tablet_copy_source_session.cc:205] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:47.568573 2936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9bbc56ba2f841b9bb36a864e1a1f808. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:47.575402 2931 tablet_bootstrap.cc:492] T b9342dbe0497499ba4c238f2ed033293 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:47.580778 31926 raft_consensus.cc:1203] T 815b1fc784714c8cb4ba76707c345625 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:26:47.582476 31389 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=b6e29d3838874defa6668514ac1c2423] alter complete (version 1)
I20240502 21:26:47.582651 2936 tablet_copy_client.cc:783] T f9bbc56ba2f841b9bb36a864e1a1f808 P 9dec645b473f40578e726f3e50e219bf: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:47.583148 2936 tablet_copy_client.cc:647] T f9bbc56ba2f841b9bb36a864e1a1f808 P 9dec645b473f40578e726f3e50e219bf: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:47.569265 2816 consensus_queue.cc:1035] T 4f09394db52e489e9bc42a4883ee8b1c P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:47.583707 2763 consensus_queue.cc:1035] T 4f09394db52e489e9bc42a4883ee8b1c P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:47.591583 2942 consensus_queue.cc:1035] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:47.596525 3022 ts_tablet_manager.cc:902] T 83904952a5a541ecac508449e23e70dd P 1ade0354c44c4c37ae480c7a94b11e1b: Initiating tablet copy from peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:47.598253 2936 tablet_copy_client.cc:523] T f9bbc56ba2f841b9bb36a864e1a1f808 P 9dec645b473f40578e726f3e50e219bf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:47.602463 2936 tablet_bootstrap.cc:492] T f9bbc56ba2f841b9bb36a864e1a1f808 P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:47.605471 2942 consensus_queue.cc:1035] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:47.620002 2506 consensus_queue.cc:1035] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:47.636261 2872 tablet_bootstrap.cc:492] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 9dec645b473f40578e726f3e50e219bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:47.660742 2872 tablet_bootstrap.cc:492] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 9dec645b473f40578e726f3e50e219bf: Bootstrap complete.
I20240502 21:26:47.661036 2872 ts_tablet_manager.cc:1367] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.190s user 0.049s sys 0.008s
I20240502 21:26:47.661998 2872 raft_consensus.cc:348] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:26:47.662287 2872 raft_consensus.cc:729] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: LEARNER
I20240502 21:26:47.662591 2872 consensus_queue.cc:260] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:26:47.663162 2872 ts_tablet_manager.cc:1398] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:47.664054 31556 tablet_copy_service.cc:342] P a44f41b0b6644fb6aa29476aed8c964f: Request end of tablet copy session 9dec645b473f40578e726f3e50e219bf-55fc18e4cc98446ca50c1dd2b6a9a89d received from {username='slave'} at 127.24.226.134:40609
I20240502 21:26:47.664263 31556 tablet_copy_service.cc:433] P a44f41b0b6644fb6aa29476aed8c964f: ending tablet copy session 9dec645b473f40578e726f3e50e219bf-55fc18e4cc98446ca50c1dd2b6a9a89d on tablet 55fc18e4cc98446ca50c1dd2b6a9a89d with peer 9dec645b473f40578e726f3e50e219bf
I20240502 21:26:47.666478 2900 raft_consensus.cc:1050] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c: attempting to promote NON_VOTER 9dec645b473f40578e726f3e50e219bf to VOTER
I20240502 21:26:47.667512 2900 consensus_queue.cc:237] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:26:47.676723 31925 raft_consensus.cc:1261] T b407c2c9b3a14ce3ada52b7856c6ecee P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:47.677701 31666 raft_consensus.cc:1261] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:47.678661 1966 raft_consensus.cc:1261] T b407c2c9b3a14ce3ada52b7856c6ecee P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:47.679406 2791 consensus_queue.cc:1035] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:47.679973 2791 consensus_queue.cc:1035] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:47.680452 2791 consensus_queue.cc:1035] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:47.697479 31395 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:26:47.698197 2506 consensus_queue.cc:1035] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.052s
I20240502 21:26:47.716773 3010 tablet.cc:1722] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:47.718643 3010 tablet.cc:1620] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d: MemRowSet was empty: no flush needed.
I20240502 21:26:47.725950 3016 tablet.cc:1722] T f332ab12c37f459ca85585657a0fe01e P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:47.727651 3016 tablet.cc:1620] T f332ab12c37f459ca85585657a0fe01e P 9dec645b473f40578e726f3e50e219bf: MemRowSet was empty: no flush needed.
I20240502 21:26:47.732969 3011 tablet.cc:1722] T f332ab12c37f459ca85585657a0fe01e P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:47.733512 3011 tablet.cc:1620] T f332ab12c37f459ca85585657a0fe01e P 1ade0354c44c4c37ae480c7a94b11e1b: MemRowSet was empty: no flush needed.
I20240502 21:26:47.742076 3007 ts_tablet_manager.cc:1888] T 49acc5301b9849e9b6795179acbc59a8 P 1ade0354c44c4c37ae480c7a94b11e1b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:26:47.747871 2791 raft_consensus.cc:2932] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:47.749183 31665 raft_consensus.cc:2932] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:47.751039 3007 log.cc:1196] T 49acc5301b9849e9b6795179acbc59a8 P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal/wals/49acc5301b9849e9b6795179acbc59a8
I20240502 21:26:47.755065 31390 catalog_manager.cc:5484] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: config changed from index 4 to 5, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:47.757475 31381 catalog_manager.cc:4830] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): tablet 49acc5301b9849e9b6795179acbc59a8 (table rebalancer_extra_table_2 [id=408b4cadaf474250a5666b6a49f4f8cc]) successfully deleted
I20240502 21:26:47.760309 31395 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=b6e29d3838874defa6668514ac1c2423] alter complete (version 2)
I20240502 21:26:47.762656 1964 raft_consensus.cc:2932] T b407c2c9b3a14ce3ada52b7856c6ecee P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:47.765331 3022 tablet_copy_client.cc:308] T 83904952a5a541ecac508449e23e70dd P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.132:35933
I20240502 21:26:47.771853 31924 raft_consensus.cc:2932] T b407c2c9b3a14ce3ada52b7856c6ecee P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:47.789959 31947 tablet_copy_service.cc:140] P 2d50f33557944bb480ea32199d9ee4f1: Received BeginTabletCopySession request for tablet 83904952a5a541ecac508449e23e70dd from peer 1ade0354c44c4c37ae480c7a94b11e1b ({username='slave'} at 127.24.226.135:33163)
I20240502 21:26:47.790138 31947 tablet_copy_service.cc:161] P 2d50f33557944bb480ea32199d9ee4f1: Beginning new tablet copy session on tablet 83904952a5a541ecac508449e23e70dd from peer 1ade0354c44c4c37ae480c7a94b11e1b at {username='slave'} at 127.24.226.135:33163: session id = 1ade0354c44c4c37ae480c7a94b11e1b-83904952a5a541ecac508449e23e70dd
I20240502 21:26:47.793156 31947 tablet_copy_source_session.cc:205] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:47.794970 3022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83904952a5a541ecac508449e23e70dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:47.801103 2931 tablet_bootstrap.cc:492] T b9342dbe0497499ba4c238f2ed033293 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:47.801744 3022 tablet_copy_client.cc:783] T 83904952a5a541ecac508449e23e70dd P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:47.802471 3022 tablet_copy_client.cc:647] T 83904952a5a541ecac508449e23e70dd P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:47.805191 3022 tablet_copy_client.cc:523] T 83904952a5a541ecac508449e23e70dd P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:47.805977 31395 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_extra_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:47.809015 3022 tablet_bootstrap.cc:492] T 83904952a5a541ecac508449e23e70dd P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:47.810688 2931 tablet_bootstrap.cc:492] T b9342dbe0497499ba4c238f2ed033293 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap complete.
I20240502 21:26:47.810978 2931 ts_tablet_manager.cc:1367] T b9342dbe0497499ba4c238f2ed033293 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.236s user 0.005s sys 0.004s
I20240502 21:26:47.816942 2931 raft_consensus.cc:348] T b9342dbe0497499ba4c238f2ed033293 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } }
I20240502 21:26:47.817231 2931 raft_consensus.cc:729] T b9342dbe0497499ba4c238f2ed033293 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: LEARNER
I20240502 21:26:47.817481 2931 consensus_queue.cc:260] T b9342dbe0497499ba4c238f2ed033293 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } }
I20240502 21:26:47.818073 2931 ts_tablet_manager.cc:1398] T b9342dbe0497499ba4c238f2ed033293 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240502 21:26:47.825323 1854 tablet_copy_service.cc:342] P 1e04157b98dd4fad98858c290b73c3db: Request end of tablet copy session 05f4fac41c4f4959b9b316cb112d0ada-b9342dbe0497499ba4c238f2ed033293 received from {username='slave'} at 127.24.226.130:43577
I20240502 21:26:47.825546 1854 tablet_copy_service.cc:433] P 1e04157b98dd4fad98858c290b73c3db: ending tablet copy session 05f4fac41c4f4959b9b316cb112d0ada-b9342dbe0497499ba4c238f2ed033293 on tablet b9342dbe0497499ba4c238f2ed033293 with peer 05f4fac41c4f4959b9b316cb112d0ada
I20240502 21:26:47.828570 1812 tablet_service.cc:1467] Processing CreateTablet for tablet dab4f06cc78946dc962267cb2aae04f7 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=112fc5f7af4e4c8080926e0a6128b468]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:47.828959 1812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dab4f06cc78946dc962267cb2aae04f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:47.829025 1946 tablet_service.cc:1467] Processing CreateTablet for tablet c5ba5610b754466fb1e12db451295809 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=112fc5f7af4e4c8080926e0a6128b468]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:47.829401 1946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5ba5610b754466fb1e12db451295809. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:47.829988 31906 tablet_service.cc:1467] Processing CreateTablet for tablet dab4f06cc78946dc962267cb2aae04f7 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=112fc5f7af4e4c8080926e0a6128b468]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:47.830386 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dab4f06cc78946dc962267cb2aae04f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:47.832723 1945 tablet_service.cc:1467] Processing CreateTablet for tablet b20c2bb60f6149458a62ee443b8cf3ab (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=112fc5f7af4e4c8080926e0a6128b468]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:47.832752 31516 tablet_service.cc:1467] Processing CreateTablet for tablet b20c2bb60f6149458a62ee443b8cf3ab (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=112fc5f7af4e4c8080926e0a6128b468]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:47.832744 31905 tablet_service.cc:1467] Processing CreateTablet for tablet c5ba5610b754466fb1e12db451295809 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=112fc5f7af4e4c8080926e0a6128b468]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:47.833063 1945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b20c2bb60f6149458a62ee443b8cf3ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:47.833091 31905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5ba5610b754466fb1e12db451295809. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:47.833122 31516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b20c2bb60f6149458a62ee443b8cf3ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:47.837450 2210 tablet_service.cc:1467] Processing CreateTablet for tablet dab4f06cc78946dc962267cb2aae04f7 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=112fc5f7af4e4c8080926e0a6128b468]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:47.837877 2210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dab4f06cc78946dc962267cb2aae04f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:47.839742 2957 tablet_bootstrap.cc:492] T 5b1dcae186f542cea92c9191fa8ef951 P d1406e92aec84ac585304e57bd37a40d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:47.840252 2957 tablet_bootstrap.cc:492] T 5b1dcae186f542cea92c9191fa8ef951 P d1406e92aec84ac585304e57bd37a40d: Bootstrap complete.
I20240502 21:26:47.840544 2957 ts_tablet_manager.cc:1367] T 5b1dcae186f542cea92c9191fa8ef951 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.381s user 0.062s sys 0.003s
I20240502 21:26:47.840746 2209 tablet_service.cc:1467] Processing CreateTablet for tablet c5ba5610b754466fb1e12db451295809 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=112fc5f7af4e4c8080926e0a6128b468]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:47.841102 2209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5ba5610b754466fb1e12db451295809. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:47.841722 2957 raft_consensus.cc:348] T 5b1dcae186f542cea92c9191fa8ef951 P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } }
I20240502 21:26:47.841990 2957 raft_consensus.cc:729] T 5b1dcae186f542cea92c9191fa8ef951 P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: LEARNER
I20240502 21:26:47.842273 2957 consensus_queue.cc:260] T 5b1dcae186f542cea92c9191fa8ef951 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } }
I20240502 21:26:47.843798 2957 ts_tablet_manager.cc:1398] T 5b1dcae186f542cea92c9191fa8ef951 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:47.844846 2208 tablet_service.cc:1467] Processing CreateTablet for tablet b20c2bb60f6149458a62ee443b8cf3ab (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=112fc5f7af4e4c8080926e0a6128b468]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:47.845019 31817 tablet_copy_service.cc:342] P 08d7a0680d244b55b93d17f9ab57b84c: Request end of tablet copy session d1406e92aec84ac585304e57bd37a40d-5b1dcae186f542cea92c9191fa8ef951 received from {username='slave'} at 127.24.226.136:57795
I20240502 21:26:47.845158 31817 tablet_copy_service.cc:433] P 08d7a0680d244b55b93d17f9ab57b84c: ending tablet copy session d1406e92aec84ac585304e57bd37a40d-5b1dcae186f542cea92c9191fa8ef951 on tablet 5b1dcae186f542cea92c9191fa8ef951 with peer d1406e92aec84ac585304e57bd37a40d
I20240502 21:26:47.845273 2208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b20c2bb60f6149458a62ee443b8cf3ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:47.849063 2967 tablet_bootstrap.cc:492] T 62f083e9641042c990efcfb1c4104de5 P d1406e92aec84ac585304e57bd37a40d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:47.849494 2967 tablet_bootstrap.cc:492] T 62f083e9641042c990efcfb1c4104de5 P d1406e92aec84ac585304e57bd37a40d: Bootstrap complete.
I20240502 21:26:47.849750 2967 ts_tablet_manager.cc:1367] T 62f083e9641042c990efcfb1c4104de5 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.344s user 0.039s sys 0.013s
I20240502 21:26:47.851966 2967 raft_consensus.cc:348] T 62f083e9641042c990efcfb1c4104de5 P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } }
I20240502 21:26:47.852211 2967 raft_consensus.cc:729] T 62f083e9641042c990efcfb1c4104de5 P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: LEARNER
I20240502 21:26:47.852392 2967 consensus_queue.cc:260] T 62f083e9641042c990efcfb1c4104de5 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } }
I20240502 21:26:47.854960 3047 tablet_bootstrap.cc:492] T c5ba5610b754466fb1e12db451295809 P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:47.855705 3048 tablet_bootstrap.cc:492] T c5ba5610b754466fb1e12db451295809 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:47.856173 2967 ts_tablet_manager.cc:1398] T 62f083e9641042c990efcfb1c4104de5 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 21:26:47.857555 3047 tablet_bootstrap.cc:654] T c5ba5610b754466fb1e12db451295809 P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:47.856890 31947 tablet_copy_service.cc:342] P 2d50f33557944bb480ea32199d9ee4f1: Request end of tablet copy session d1406e92aec84ac585304e57bd37a40d-62f083e9641042c990efcfb1c4104de5 received from {username='slave'} at 127.24.226.136:43115
I20240502 21:26:47.858182 31947 tablet_copy_service.cc:433] P 2d50f33557944bb480ea32199d9ee4f1: ending tablet copy session d1406e92aec84ac585304e57bd37a40d-62f083e9641042c990efcfb1c4104de5 on tablet 62f083e9641042c990efcfb1c4104de5 with peer d1406e92aec84ac585304e57bd37a40d
I20240502 21:26:47.865286 3048 tablet_bootstrap.cc:654] T c5ba5610b754466fb1e12db451295809 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:47.869429 3047 tablet_bootstrap.cc:492] T c5ba5610b754466fb1e12db451295809 P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:26:47.869597 3047 ts_tablet_manager.cc:1367] T c5ba5610b754466fb1e12db451295809 P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.002s
I20240502 21:26:47.870460 3047 raft_consensus.cc:348] T c5ba5610b754466fb1e12db451295809 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:47.870754 3047 raft_consensus.cc:374] T c5ba5610b754466fb1e12db451295809 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:47.870921 3047 raft_consensus.cc:729] T c5ba5610b754466fb1e12db451295809 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:26:47.870975 31393 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:26:47.871461 3047 consensus_queue.cc:260] T c5ba5610b754466fb1e12db451295809 P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:47.872752 3047 ts_tablet_manager.cc:1398] T c5ba5610b754466fb1e12db451295809 P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:26:47.873050 3047 tablet_bootstrap.cc:492] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:47.877245 3047 tablet_bootstrap.cc:654] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:47.892824 3010 tablet.cc:1722] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:47.892997 3011 tablet.cc:1722] T f332ab12c37f459ca85585657a0fe01e P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:47.899462 3016 tablet.cc:1722] T f332ab12c37f459ca85585657a0fe01e P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:47.902413 31390 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table_new_name [id=b6e29d3838874defa6668514ac1c2423] alter complete (version 3)
I20240502 21:26:47.912734 3050 tablet_bootstrap.cc:492] T b20c2bb60f6149458a62ee443b8cf3ab P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:47.915195 3050 tablet_bootstrap.cc:654] T b20c2bb60f6149458a62ee443b8cf3ab P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:47.918663 3052 tablet_bootstrap.cc:492] T dab4f06cc78946dc962267cb2aae04f7 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:47.924683 3002 raft_consensus.cc:482] T c5ba5610b754466fb1e12db451295809 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:47.924726 3053 tablet_bootstrap.cc:492] T dab4f06cc78946dc962267cb2aae04f7 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:47.924829 3002 raft_consensus.cc:504] T c5ba5610b754466fb1e12db451295809 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:47.925593 3002 leader_election.cc:288] T c5ba5610b754466fb1e12db451295809 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:47.926285 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5ba5610b754466fb1e12db451295809" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:47.926384 3052 tablet_bootstrap.cc:654] T dab4f06cc78946dc962267cb2aae04f7 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
W20240502 21:26:47.926908 1902 leader_election.cc:341] T c5ba5610b754466fb1e12db451295809 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:47.927207 3048 tablet_bootstrap.cc:492] T c5ba5610b754466fb1e12db451295809 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:47.927367 3048 ts_tablet_manager.cc:1367] T c5ba5610b754466fb1e12db451295809 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.072s user 0.011s sys 0.006s
I20240502 21:26:47.930740 2230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5ba5610b754466fb1e12db451295809" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:26:47.930891 3052 tablet_bootstrap.cc:492] T dab4f06cc78946dc962267cb2aae04f7 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:47.931027 3052 ts_tablet_manager.cc:1367] T dab4f06cc78946dc962267cb2aae04f7 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
W20240502 21:26:47.933387 1901 leader_election.cc:341] T c5ba5610b754466fb1e12db451295809 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:47.933573 1901 leader_election.cc:302] T c5ba5610b754466fb1e12db451295809 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dec645b473f40578e726f3e50e219bf; no voters: 2d50f33557944bb480ea32199d9ee4f1, d1406e92aec84ac585304e57bd37a40d
I20240502 21:26:47.933931 2936 tablet_bootstrap.cc:492] T f9bbc56ba2f841b9bb36a864e1a1f808 P 9dec645b473f40578e726f3e50e219bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:47.934180 3048 raft_consensus.cc:348] T c5ba5610b754466fb1e12db451295809 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:47.934458 3053 tablet_bootstrap.cc:654] T dab4f06cc78946dc962267cb2aae04f7 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:47.934831 3002 raft_consensus.cc:2726] T c5ba5610b754466fb1e12db451295809 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:26:47.934969 3048 raft_consensus.cc:374] T c5ba5610b754466fb1e12db451295809 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:47.935088 3048 raft_consensus.cc:729] T c5ba5610b754466fb1e12db451295809 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:47.934880 2936 tablet_bootstrap.cc:492] T f9bbc56ba2f841b9bb36a864e1a1f808 P 9dec645b473f40578e726f3e50e219bf: Bootstrap complete.
I20240502 21:26:47.935444 3048 consensus_queue.cc:260] T c5ba5610b754466fb1e12db451295809 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:47.935953 2936 ts_tablet_manager.cc:1367] T f9bbc56ba2f841b9bb36a864e1a1f808 P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.334s user 0.051s sys 0.003s
I20240502 21:26:47.936661 3052 raft_consensus.cc:348] T dab4f06cc78946dc962267cb2aae04f7 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:47.936774 2936 raft_consensus.cc:348] T f9bbc56ba2f841b9bb36a864e1a1f808 P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:26:47.936909 3052 raft_consensus.cc:374] T dab4f06cc78946dc962267cb2aae04f7 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:47.937002 3052 raft_consensus.cc:729] T dab4f06cc78946dc962267cb2aae04f7 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:47.937294 3052 consensus_queue.cc:260] T dab4f06cc78946dc962267cb2aae04f7 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:47.938637 3052 ts_tablet_manager.cc:1398] T dab4f06cc78946dc962267cb2aae04f7 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 21:26:47.940526 3050 tablet_bootstrap.cc:492] T b20c2bb60f6149458a62ee443b8cf3ab P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:47.940711 3050 ts_tablet_manager.cc:1367] T b20c2bb60f6149458a62ee443b8cf3ab P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.000s
I20240502 21:26:47.941146 3048 ts_tablet_manager.cc:1398] T c5ba5610b754466fb1e12db451295809 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20240502 21:26:47.941547 3050 raft_consensus.cc:348] T b20c2bb60f6149458a62ee443b8cf3ab P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:47.941910 3050 raft_consensus.cc:374] T b20c2bb60f6149458a62ee443b8cf3ab P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:47.942054 3050 raft_consensus.cc:729] T b20c2bb60f6149458a62ee443b8cf3ab P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:47.942392 3048 tablet_bootstrap.cc:492] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:47.942330 3050 consensus_queue.cc:260] T b20c2bb60f6149458a62ee443b8cf3ab P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:47.943223 2936 raft_consensus.cc:729] T f9bbc56ba2f841b9bb36a864e1a1f808 P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: LEARNER
I20240502 21:26:47.943547 2936 consensus_queue.cc:260] T f9bbc56ba2f841b9bb36a864e1a1f808 P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:26:47.945156 3048 tablet_bootstrap.cc:654] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:47.946372 2936 ts_tablet_manager.cc:1398] T f9bbc56ba2f841b9bb36a864e1a1f808 P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 21:26:47.947482 3050 ts_tablet_manager.cc:1398] T b20c2bb60f6149458a62ee443b8cf3ab P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 21:26:47.948406 3047 tablet_bootstrap.cc:492] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:26:47.950999 3047 ts_tablet_manager.cc:1367] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.078s user 0.017s sys 0.019s
I20240502 21:26:47.951766 31817 tablet_copy_service.cc:342] P 08d7a0680d244b55b93d17f9ab57b84c: Request end of tablet copy session 9dec645b473f40578e726f3e50e219bf-f9bbc56ba2f841b9bb36a864e1a1f808 received from {username='slave'} at 127.24.226.134:42809
I20240502 21:26:47.951752 3047 raft_consensus.cc:348] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:47.951890 31817 tablet_copy_service.cc:433] P 08d7a0680d244b55b93d17f9ab57b84c: ending tablet copy session 9dec645b473f40578e726f3e50e219bf-f9bbc56ba2f841b9bb36a864e1a1f808 on tablet f9bbc56ba2f841b9bb36a864e1a1f808 with peer 9dec645b473f40578e726f3e50e219bf
I20240502 21:26:47.951990 3047 raft_consensus.cc:374] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:47.952055 3047 raft_consensus.cc:729] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:26:47.952315 3047 consensus_queue.cc:260] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:47.952797 3047 ts_tablet_manager.cc:1398] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:47.962656 3048 tablet_bootstrap.cc:492] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:47.962880 3048 ts_tablet_manager.cc:1367] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.002s
I20240502 21:26:47.963687 3048 raft_consensus.cc:348] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:47.964015 3048 raft_consensus.cc:374] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:47.964164 3048 raft_consensus.cc:729] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:47.964579 3053 tablet_bootstrap.cc:492] T dab4f06cc78946dc962267cb2aae04f7 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:47.964548 3048 consensus_queue.cc:260] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:47.964789 3053 ts_tablet_manager.cc:1367] T dab4f06cc78946dc962267cb2aae04f7 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.000s
I20240502 21:26:47.966146 3048 ts_tablet_manager.cc:1398] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:47.967118 2944 tablet_bootstrap.cc:492] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:47.967204 3053 raft_consensus.cc:348] T dab4f06cc78946dc962267cb2aae04f7 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:47.967548 2944 tablet_bootstrap.cc:492] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap complete.
I20240502 21:26:47.967664 3053 raft_consensus.cc:374] T dab4f06cc78946dc962267cb2aae04f7 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:47.967828 2944 ts_tablet_manager.cc:1367] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.412s user 0.045s sys 0.016s
I20240502 21:26:47.968048 3053 raft_consensus.cc:729] T dab4f06cc78946dc962267cb2aae04f7 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:47.968374 3053 consensus_queue.cc:260] T dab4f06cc78946dc962267cb2aae04f7 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:47.968562 2944 raft_consensus.cc:348] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } }
I20240502 21:26:47.968838 2944 raft_consensus.cc:729] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: LEARNER
I20240502 21:26:47.969254 2944 consensus_queue.cc:260] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } }
I20240502 21:26:47.969900 2944 ts_tablet_manager.cc:1398] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 21:26:47.970587 31817 tablet_copy_service.cc:342] P 08d7a0680d244b55b93d17f9ab57b84c: Request end of tablet copy session 1e04157b98dd4fad98858c290b73c3db-9ea8ddb9f8e04893b56c10dbbb5af321 received from {username='slave'} at 127.24.226.133:41593
I20240502 21:26:47.970700 3002 raft_consensus.cc:482] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:47.970728 31817 tablet_copy_service.cc:433] P 08d7a0680d244b55b93d17f9ab57b84c: ending tablet copy session 1e04157b98dd4fad98858c290b73c3db-9ea8ddb9f8e04893b56c10dbbb5af321 on tablet 9ea8ddb9f8e04893b56c10dbbb5af321 with peer 1e04157b98dd4fad98858c290b73c3db
I20240502 21:26:47.970839 3002 raft_consensus.cc:504] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:47.971449 3002 leader_election.cc:288] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:47.973512 2228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b20c2bb60f6149458a62ee443b8cf3ab" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:26:47.973659 31534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b20c2bb60f6149458a62ee443b8cf3ab" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:47.973870 31534 raft_consensus.cc:2454] T b20c2bb60f6149458a62ee443b8cf3ab P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 0.
I20240502 21:26:47.974337 1902 leader_election.cc:302] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dec645b473f40578e726f3e50e219bf, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:47.974654 3002 raft_consensus.cc:2781] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:47.974756 3002 raft_consensus.cc:482] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:47.974848 3002 raft_consensus.cc:3037] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:47.976753 3053 ts_tablet_manager.cc:1398] T dab4f06cc78946dc962267cb2aae04f7 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
W20240502 21:26:47.977900 1901 leader_election.cc:341] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:47.977270 3002 raft_consensus.cc:504] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:47.979017 31534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b20c2bb60f6149458a62ee443b8cf3ab" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:26:47.979211 31534 raft_consensus.cc:3037] T b20c2bb60f6149458a62ee443b8cf3ab P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:47.981175 3053 tablet_bootstrap.cc:492] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:47.981777 31534 raft_consensus.cc:2454] T b20c2bb60f6149458a62ee443b8cf3ab P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 1.
I20240502 21:26:47.982762 3053 tablet_bootstrap.cc:654] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:47.985455 2228 raft_consensus.cc:1203] T 5b1dcae186f542cea92c9191fa8ef951 P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:26:47.990410 1966 raft_consensus.cc:1203] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:26:47.991075 1902 leader_election.cc:302] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dec645b473f40578e726f3e50e219bf, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:47.992164 31666 raft_consensus.cc:1203] T b9342dbe0497499ba4c238f2ed033293 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:26:47.998816 3002 leader_election.cc:288] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 election: Requested vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:47.998811 2228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b20c2bb60f6149458a62ee443b8cf3ab" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
W20240502 21:26:47.999293 1901 leader_election.cc:341] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:47.999460 3067 raft_consensus.cc:2781] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:47.999577 3067 raft_consensus.cc:686] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf [term 1 LEADER]: Becoming Leader. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Running, Role: LEADER
I20240502 21:26:47.999806 3067 consensus_queue.cc:237] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:48.001336 3053 tablet_bootstrap.cc:492] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:48.001529 3053 ts_tablet_manager.cc:1367] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20240502 21:26:48.002184 3053 raft_consensus.cc:348] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:48.002416 3053 raft_consensus.cc:374] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:48.002511 3053 raft_consensus.cc:729] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:48.002753 3053 consensus_queue.cc:260] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:48.004174 3053 ts_tablet_manager.cc:1398] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:26:48.005228 31391 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:48.005468 31391 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:48.009178 31391 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet f332ab12c37f459ca85585657a0fe01e
I20240502 21:26:48.010579 1946 tablet_service.cc:1514] Processing DeleteTablet for tablet f332ab12c37f459ca85585657a0fe01e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:48 UTC) from {username='slave'} at 127.0.0.1:33002
I20240502 21:26:48.011132 2210 tablet_service.cc:1514] Processing DeleteTablet for tablet f332ab12c37f459ca85585657a0fe01e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:48 UTC) from {username='slave'} at 127.0.0.1:36984
I20240502 21:26:48.011596 2075 tablet_service.cc:1514] Processing DeleteTablet for tablet f332ab12c37f459ca85585657a0fe01e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:48 UTC) from {username='slave'} at 127.0.0.1:36740
I20240502 21:26:48.011929 3007 tablet_replica.cc:330] T f332ab12c37f459ca85585657a0fe01e P 1ade0354c44c4c37ae480c7a94b11e1b: stopping tablet replica
I20240502 21:26:48.012169 3007 raft_consensus.cc:2229] T f332ab12c37f459ca85585657a0fe01e P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:48.012418 3007 raft_consensus.cc:2258] T f332ab12c37f459ca85585657a0fe01e P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:48.012952 3053 tablet_bootstrap.cc:492] T b20c2bb60f6149458a62ee443b8cf3ab P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:48.014714 3007 ts_tablet_manager.cc:1875] T f332ab12c37f459ca85585657a0fe01e P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:48.021876 3053 tablet_bootstrap.cc:654] T b20c2bb60f6149458a62ee443b8cf3ab P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:48.023103 3070 tablet_replica.cc:330] T f332ab12c37f459ca85585657a0fe01e P 9dec645b473f40578e726f3e50e219bf: stopping tablet replica
I20240502 21:26:48.023288 3070 raft_consensus.cc:2229] T f332ab12c37f459ca85585657a0fe01e P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:48.023535 3070 raft_consensus.cc:2258] T f332ab12c37f459ca85585657a0fe01e P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:48.025787 31391 catalog_manager.cc:5484] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dec645b473f40578e726f3e50e219bf (127.24.226.134). New cstate: current_term: 1 leader_uuid: "9dec645b473f40578e726f3e50e219bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:48.032253 3071 tablet_replica.cc:330] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d: stopping tablet replica
I20240502 21:26:48.036623 3071 raft_consensus.cc:2229] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:48.037159 3071 raft_consensus.cc:2258] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:48.039469 3071 ts_tablet_manager.cc:1875] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:48.040948 3007 ts_tablet_manager.cc:1888] T f332ab12c37f459ca85585657a0fe01e P 1ade0354c44c4c37ae480c7a94b11e1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:48.041759 3070 ts_tablet_manager.cc:1875] T f332ab12c37f459ca85585657a0fe01e P 9dec645b473f40578e726f3e50e219bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:48.042098 3007 log.cc:1196] T f332ab12c37f459ca85585657a0fe01e P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal/wals/f332ab12c37f459ca85585657a0fe01e
I20240502 21:26:48.043119 3007 ts_tablet_manager.cc:1909] T f332ab12c37f459ca85585657a0fe01e P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting consensus metadata
I20240502 21:26:48.047484 31381 catalog_manager.cc:4830] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): tablet f332ab12c37f459ca85585657a0fe01e (table rebalancer_dynamic_table_new_name [id=b6e29d3838874defa6668514ac1c2423]) successfully deleted
I20240502 21:26:48.053256 3053 tablet_bootstrap.cc:492] T b20c2bb60f6149458a62ee443b8cf3ab P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:48.053524 3053 ts_tablet_manager.cc:1367] T b20c2bb60f6149458a62ee443b8cf3ab P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.041s user 0.006s sys 0.000s
I20240502 21:26:48.054356 3053 raft_consensus.cc:348] T b20c2bb60f6149458a62ee443b8cf3ab P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:48.054716 3053 raft_consensus.cc:374] T b20c2bb60f6149458a62ee443b8cf3ab P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:48.054826 3053 raft_consensus.cc:729] T b20c2bb60f6149458a62ee443b8cf3ab P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:48.055123 3053 consensus_queue.cc:260] T b20c2bb60f6149458a62ee443b8cf3ab P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:48.058238 3053 ts_tablet_manager.cc:1398] T b20c2bb60f6149458a62ee443b8cf3ab P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:26:48.059134 2791 raft_consensus.cc:1050] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c: attempting to promote NON_VOTER 2d50f33557944bb480ea32199d9ee4f1 to VOTER
I20240502 21:26:48.060128 2791 consensus_queue.cc:237] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } }
I20240502 21:26:48.061340 2228 raft_consensus.cc:1203] T 62f083e9641042c990efcfb1c4104de5 P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:26:48.067505 31534 raft_consensus.cc:1261] T 815b1fc784714c8cb4ba76707c345625 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:48.067541 1833 raft_consensus.cc:1261] T 815b1fc784714c8cb4ba76707c345625 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:48.068109 2900 consensus_queue.cc:1035] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:48.068892 31925 raft_consensus.cc:1261] T 815b1fc784714c8cb4ba76707c345625 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEARNER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:48.069331 3033 consensus_queue.cc:1035] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:48.069927 2900 consensus_queue.cc:1035] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:48.070590 3071 ts_tablet_manager.cc:1888] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:48.070739 3071 log.cc:1196] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal/wals/f332ab12c37f459ca85585657a0fe01e
I20240502 21:26:48.071213 3071 ts_tablet_manager.cc:1909] T f332ab12c37f459ca85585657a0fe01e P d1406e92aec84ac585304e57bd37a40d: Deleting consensus metadata
I20240502 21:26:48.072232 31381 catalog_manager.cc:4830] TS d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): tablet f332ab12c37f459ca85585657a0fe01e (table rebalancer_dynamic_table_new_name [id=b6e29d3838874defa6668514ac1c2423]) successfully deleted
I20240502 21:26:48.082270 3070 ts_tablet_manager.cc:1888] T f332ab12c37f459ca85585657a0fe01e P 9dec645b473f40578e726f3e50e219bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:48.082746 3070 log.cc:1196] T f332ab12c37f459ca85585657a0fe01e P 9dec645b473f40578e726f3e50e219bf: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal/wals/f332ab12c37f459ca85585657a0fe01e
I20240502 21:26:48.085937 3070 ts_tablet_manager.cc:1909] T f332ab12c37f459ca85585657a0fe01e P 9dec645b473f40578e726f3e50e219bf: Deleting consensus metadata
I20240502 21:26:48.088038 31379 catalog_manager.cc:4830] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): tablet f332ab12c37f459ca85585657a0fe01e (table rebalancer_dynamic_table_new_name [id=b6e29d3838874defa6668514ac1c2423]) successfully deleted
I20240502 21:26:48.092844 3022 tablet_bootstrap.cc:492] T 83904952a5a541ecac508449e23e70dd P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:48.093230 3022 tablet_bootstrap.cc:492] T 83904952a5a541ecac508449e23e70dd P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap complete.
I20240502 21:26:48.093498 3022 ts_tablet_manager.cc:1367] T 83904952a5a541ecac508449e23e70dd P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.285s user 0.046s sys 0.006s
I20240502 21:26:48.094302 3022 raft_consensus.cc:348] T 83904952a5a541ecac508449e23e70dd P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } }
I20240502 21:26:48.094576 3022 raft_consensus.cc:729] T 83904952a5a541ecac508449e23e70dd P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: LEARNER
I20240502 21:26:48.094786 3022 consensus_queue.cc:260] T 83904952a5a541ecac508449e23e70dd P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } }
I20240502 21:26:48.095300 3022 ts_tablet_manager.cc:1398] T 83904952a5a541ecac508449e23e70dd P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:26:48.096913 31947 tablet_copy_service.cc:342] P 2d50f33557944bb480ea32199d9ee4f1: Request end of tablet copy session 1ade0354c44c4c37ae480c7a94b11e1b-83904952a5a541ecac508449e23e70dd received from {username='slave'} at 127.24.226.135:33163
I20240502 21:26:48.097044 31947 tablet_copy_service.cc:433] P 2d50f33557944bb480ea32199d9ee4f1: ending tablet copy session 1ade0354c44c4c37ae480c7a94b11e1b-83904952a5a541ecac508449e23e70dd on tablet 83904952a5a541ecac508449e23e70dd with peer 1ade0354c44c4c37ae480c7a94b11e1b
I20240502 21:26:48.100468 3033 raft_consensus.cc:2932] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:26:48.103883 1833 raft_consensus.cc:2932] T 815b1fc784714c8cb4ba76707c345625 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:26:48.105482 31925 raft_consensus.cc:2932] T 815b1fc784714c8cb4ba76707c345625 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:26:48.106298 31391 catalog_manager.cc:5484] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index 2 to 3, 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:48.110344 31534 raft_consensus.cc:2932] T 815b1fc784714c8cb4ba76707c345625 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:26:48.112531 31795 consensus_queue.cc:237] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } }
I20240502 21:26:48.114485 31389 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:26:48.115590 31925 raft_consensus.cc:1261] T 815b1fc784714c8cb4ba76707c345625 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:48.116675 2791 consensus_queue.cc:1035] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:48.123641 1946 tablet_service.cc:1467] Processing CreateTablet for tablet 5019de9e275c42a697d17fc31bceac36 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=df46a59a7eee43beba75e74228f448ac]), partition=
I20240502 21:26:48.124034 1946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5019de9e275c42a697d17fc31bceac36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:48.124200 3033 raft_consensus.cc:2932] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:26:48.124709 31535 raft_consensus.cc:1261] T 815b1fc784714c8cb4ba76707c345625 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:48.126471 31647 tablet_service.cc:1467] Processing CreateTablet for tablet 5019de9e275c42a697d17fc31bceac36 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=df46a59a7eee43beba75e74228f448ac]), partition=
I20240502 21:26:48.126673 31516 tablet_service.cc:1467] Processing CreateTablet for tablet 5019de9e275c42a697d17fc31bceac36 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=df46a59a7eee43beba75e74228f448ac]), partition=
I20240502 21:26:48.126868 31647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5019de9e275c42a697d17fc31bceac36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:48.126986 31516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5019de9e275c42a697d17fc31bceac36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:48.135321 31381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 815b1fc784714c8cb4ba76707c345625 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:48.136118 2777 raft_consensus.cc:482] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:48.136267 2777 raft_consensus.cc:504] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:48.136919 2777 leader_election.cc:288] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:48.138720 2799 consensus_queue.cc:1035] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:48.140344 31925 raft_consensus.cc:2932] T 815b1fc784714c8cb4ba76707c345625 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:26:48.140847 2228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dab4f06cc78946dc962267cb2aae04f7" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:26:48.141072 2228 raft_consensus.cc:2454] T dab4f06cc78946dc962267cb2aae04f7 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:48.144977 31862 leader_election.cc:302] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:48.145643 1884 raft_consensus.cc:2781] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:48.145751 1884 raft_consensus.cc:482] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:48.145833 1884 raft_consensus.cc:3037] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:48.146939 1833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dab4f06cc78946dc962267cb2aae04f7" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:26:48.147140 1833 raft_consensus.cc:2454] T dab4f06cc78946dc962267cb2aae04f7 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:48.150535 3082 tablet_bootstrap.cc:492] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:48.150369 31391 catalog_manager.cc:5484] T 815b1fc784714c8cb4ba76707c345625 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: config changed from index 3 to 4, VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) evicted. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:26:48.152307 3082 tablet_bootstrap.cc:654] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:48.154279 1812 tablet_service.cc:1514] Processing DeleteTablet for tablet 815b1fc784714c8cb4ba76707c345625 with delete_type TABLET_DATA_TOMBSTONED (TS 1e04157b98dd4fad98858c290b73c3db not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47510
I20240502 21:26:48.157233 1884 raft_consensus.cc:504] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:48.157824 1884 leader_election.cc:288] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Requested vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:48.158740 2228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dab4f06cc78946dc962267cb2aae04f7" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:26:48.158946 2228 raft_consensus.cc:3037] T dab4f06cc78946dc962267cb2aae04f7 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:48.161952 2228 raft_consensus.cc:2454] T dab4f06cc78946dc962267cb2aae04f7 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:26:48.167105 1834 raft_consensus.cc:1203] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:26:48.171439 31862 leader_election.cc:302] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:48.177490 2777 raft_consensus.cc:2781] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:48.178571 1833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dab4f06cc78946dc962267cb2aae04f7" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:26:48.178884 1833 raft_consensus.cc:3037] T dab4f06cc78946dc962267cb2aae04f7 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:48.182348 31535 raft_consensus.cc:2932] T 815b1fc784714c8cb4ba76707c345625 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:26:48.184811 3050 tablet_bootstrap.cc:492] T 5019de9e275c42a697d17fc31bceac36 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:48.190476 3050 tablet_bootstrap.cc:654] T 5019de9e275c42a697d17fc31bceac36 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:48.193622 2777 raft_consensus.cc:686] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Running, Role: LEADER
I20240502 21:26:48.194046 2777 consensus_queue.cc:237] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:48.211364 2891 raft_consensus.cc:482] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:48.211556 3047 tablet_bootstrap.cc:492] T 5019de9e275c42a697d17fc31bceac36 P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:48.211517 2891 raft_consensus.cc:504] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:48.212451 2891 leader_election.cc:288] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:48.213232 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5ba5610b754466fb1e12db451295809" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:48.213518 31925 raft_consensus.cc:2454] T c5ba5610b754466fb1e12db451295809 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:26:48.213676 3047 tablet_bootstrap.cc:654] T 5019de9e275c42a697d17fc31bceac36 P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:48.214195 2164 leader_election.cc:302] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:48.214854 2891 raft_consensus.cc:2781] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:48.214980 2891 raft_consensus.cc:482] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:48.215068 2891 raft_consensus.cc:3037] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:48.215075 1966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5ba5610b754466fb1e12db451295809" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:26:48.215330 1966 raft_consensus.cc:2454] T c5ba5610b754466fb1e12db451295809 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:26:48.217276 2429 raft_consensus.cc:482] T b20c2bb60f6149458a62ee443b8cf3ab P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:48.217468 2429 raft_consensus.cc:504] T b20c2bb60f6149458a62ee443b8cf3ab P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:48.218331 2429 leader_election.cc:288] T b20c2bb60f6149458a62ee443b8cf3ab P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:26:48.218426 2891 raft_consensus.cc:504] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:48.219136 2891 leader_election.cc:288] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Requested vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:48.221873 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5ba5610b754466fb1e12db451295809" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:48.222072 31925 raft_consensus.cc:3037] T c5ba5610b754466fb1e12db451295809 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:48.225100 1965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5ba5610b754466fb1e12db451295809" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf"
I20240502 21:26:48.225382 1965 raft_consensus.cc:3037] T c5ba5610b754466fb1e12db451295809 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:48.227993 1965 raft_consensus.cc:2454] T c5ba5610b754466fb1e12db451295809 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 1.
I20240502 21:26:48.229243 1966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b20c2bb60f6149458a62ee443b8cf3ab" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:26:48.229912 2161 leader_election.cc:302] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dec645b473f40578e726f3e50e219bf, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:48.230432 2891 raft_consensus.cc:2781] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:48.230579 2891 raft_consensus.cc:686] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Becoming Leader. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Running, Role: LEADER
I20240502 21:26:48.230844 2891 consensus_queue.cc:237] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:48.233812 31925 raft_consensus.cc:2454] T c5ba5610b754466fb1e12db451295809 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 1.
I20240502 21:26:48.235037 31389 catalog_manager.cc:5484] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d reported cstate change: term changed from 0 to 1, leader changed from <none> to d1406e92aec84ac585304e57bd37a40d (127.24.226.136). New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:48.237040 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b20c2bb60f6149458a62ee443b8cf3ab" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:48.237390 31536 raft_consensus.cc:2379] T b20c2bb60f6149458a62ee443b8cf3ab P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1406e92aec84ac585304e57bd37a40d in current term 1: Already voted for candidate 9dec645b473f40578e726f3e50e219bf in this term.
I20240502 21:26:48.237886 2164 leader_election.cc:302] T b20c2bb60f6149458a62ee443b8cf3ab P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1406e92aec84ac585304e57bd37a40d; no voters: 9dec645b473f40578e726f3e50e219bf, a44f41b0b6644fb6aa29476aed8c964f
I20240502 21:26:48.238986 2891 raft_consensus.cc:3037] T b20c2bb60f6149458a62ee443b8cf3ab P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:48.241809 2891 raft_consensus.cc:2726] T b20c2bb60f6149458a62ee443b8cf3ab P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:26:48.249785 1833 raft_consensus.cc:2454] T dab4f06cc78946dc962267cb2aae04f7 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:26:48.277567 31389 catalog_manager.cc:5484] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132). New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:48.279851 3050 tablet_bootstrap.cc:492] T 5019de9e275c42a697d17fc31bceac36 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:48.280860 3050 ts_tablet_manager.cc:1367] T 5019de9e275c42a697d17fc31bceac36 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.095s user 0.002s sys 0.011s
I20240502 21:26:48.281901 3050 raft_consensus.cc:348] T 5019de9e275c42a697d17fc31bceac36 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:48.282166 3050 raft_consensus.cc:374] T 5019de9e275c42a697d17fc31bceac36 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:48.282256 3050 raft_consensus.cc:729] T 5019de9e275c42a697d17fc31bceac36 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:48.282526 3050 consensus_queue.cc:260] T 5019de9e275c42a697d17fc31bceac36 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:48.283286 3084 tablet_replica.cc:330] T 815b1fc784714c8cb4ba76707c345625 P 1e04157b98dd4fad98858c290b73c3db: stopping tablet replica
I20240502 21:26:48.283571 3084 raft_consensus.cc:2229] T 815b1fc784714c8cb4ba76707c345625 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:48.283912 3084 raft_consensus.cc:2258] T 815b1fc784714c8cb4ba76707c345625 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:48.286882 3050 ts_tablet_manager.cc:1398] T 5019de9e275c42a697d17fc31bceac36 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 21:26:48.288493 3047 tablet_bootstrap.cc:492] T 5019de9e275c42a697d17fc31bceac36 P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:26:48.288861 3047 ts_tablet_manager.cc:1367] T 5019de9e275c42a697d17fc31bceac36 P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.077s user 0.001s sys 0.011s
I20240502 21:26:48.291131 3047 raft_consensus.cc:348] T 5019de9e275c42a697d17fc31bceac36 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:48.291388 3047 raft_consensus.cc:374] T 5019de9e275c42a697d17fc31bceac36 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:48.291501 3047 raft_consensus.cc:729] T 5019de9e275c42a697d17fc31bceac36 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:26:48.291783 3047 consensus_queue.cc:260] T 5019de9e275c42a697d17fc31bceac36 P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:48.292495 3047 ts_tablet_manager.cc:1398] T 5019de9e275c42a697d17fc31bceac36 P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:26:48.294214 3084 ts_tablet_manager.cc:1875] T 815b1fc784714c8cb4ba76707c345625 P 1e04157b98dd4fad98858c290b73c3db: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:48.294745 3082 tablet_bootstrap.cc:492] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:48.294929 3082 ts_tablet_manager.cc:1367] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.144s user 0.032s sys 0.075s
I20240502 21:26:48.295763 3082 raft_consensus.cc:348] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:48.295996 3082 raft_consensus.cc:374] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:48.296123 3082 raft_consensus.cc:729] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:48.296360 3082 consensus_queue.cc:260] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:48.297037 3082 ts_tablet_manager.cc:1398] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:48.301980 3084 ts_tablet_manager.cc:1888] T 815b1fc784714c8cb4ba76707c345625 P 1e04157b98dd4fad98858c290b73c3db: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:26:48.302408 3084 log.cc:1196] T 815b1fc784714c8cb4ba76707c345625 P 1e04157b98dd4fad98858c290b73c3db: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal/wals/815b1fc784714c8cb4ba76707c345625
I20240502 21:26:48.303172 31381 catalog_manager.cc:4830] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): tablet 815b1fc784714c8cb4ba76707c345625 (table rebalancer_extra_table_0 [id=9e962597ef7c41f493f1a41cd94c913b]) successfully deleted
I20240502 21:26:48.335904 1966 raft_consensus.cc:1203] T f9bbc56ba2f841b9bb36a864e1a1f808 P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:26:48.383412 2816 raft_consensus.cc:1050] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db: attempting to promote NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada to VOTER
I20240502 21:26:48.384850 2816 consensus_queue.cc:237] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } }
I20240502 21:26:48.387893 31795 raft_consensus.cc:1261] T b9342dbe0497499ba4c238f2ed033293 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:48.387921 31666 raft_consensus.cc:1261] T b9342dbe0497499ba4c238f2ed033293 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:48.388453 2095 raft_consensus.cc:1261] T b9342dbe0497499ba4c238f2ed033293 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:48.389381 2763 consensus_queue.cc:1035] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:48.389705 3137 consensus_queue.cc:1035] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:48.390467 3066 consensus_queue.cc:1035] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:48.394757 2763 raft_consensus.cc:2932] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:26:48.396925 31795 raft_consensus.cc:2932] T b9342dbe0497499ba4c238f2ed033293 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:26:48.399168 31666 raft_consensus.cc:2932] T b9342dbe0497499ba4c238f2ed033293 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:26:48.401155 31389 catalog_manager.cc:5484] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db reported cstate change: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:48.405781 1833 consensus_queue.cc:237] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } }
I20240502 21:26:48.407606 31665 raft_consensus.cc:1261] T b9342dbe0497499ba4c238f2ed033293 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:48.408142 3137 consensus_queue.cc:1035] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:48.410918 31795 raft_consensus.cc:1261] T b9342dbe0497499ba4c238f2ed033293 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:48.411454 3137 consensus_queue.cc:1035] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:48.411834 3066 raft_consensus.cc:2932] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:26:48.412417 31665 raft_consensus.cc:2932] T b9342dbe0497499ba4c238f2ed033293 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:26:48.414923 31381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b9342dbe0497499ba4c238f2ed033293 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:48.415585 31795 raft_consensus.cc:2932] T b9342dbe0497499ba4c238f2ed033293 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:26:48.416669 31389 catalog_manager.cc:5484] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db reported cstate change: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:48.422473 2075 tablet_service.cc:1514] Processing DeleteTablet for tablet b9342dbe0497499ba4c238f2ed033293 with delete_type TABLET_DATA_TOMBSTONED (TS 1ade0354c44c4c37ae480c7a94b11e1b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36740
I20240502 21:26:48.422796 3007 tablet_replica.cc:330] T b9342dbe0497499ba4c238f2ed033293 P 1ade0354c44c4c37ae480c7a94b11e1b: stopping tablet replica
I20240502 21:26:48.423018 3007 raft_consensus.cc:2229] T b9342dbe0497499ba4c238f2ed033293 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:48.423219 3007 pending_rounds.cc:62] T b9342dbe0497499ba4c238f2ed033293 P 1ade0354c44c4c37ae480c7a94b11e1b: Trying to abort 1 pending ops.
I20240502 21:26:48.423363 3007 pending_rounds.cc:68] T b9342dbe0497499ba4c238f2ed033293 P 1ade0354c44c4c37ae480c7a94b11e1b: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7023350613540831232 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "b9342dbe0497499ba4c238f2ed033293" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } } }
I20240502 21:26:48.423856 3007 raft_consensus.cc:2894] T b9342dbe0497499ba4c238f2ed033293 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240502 21:26:48.424018 3007 raft_consensus.cc:2866] T b9342dbe0497499ba4c238f2ed033293 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 21:26:48.424144 3007 raft_consensus.cc:2258] T b9342dbe0497499ba4c238f2ed033293 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:48.424999 3007 ts_tablet_manager.cc:1875] T b9342dbe0497499ba4c238f2ed033293 P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:48.429831 3007 ts_tablet_manager.cc:1888] T b9342dbe0497499ba4c238f2ed033293 P 1ade0354c44c4c37ae480c7a94b11e1b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:26:48.429961 3007 log.cc:1196] T b9342dbe0497499ba4c238f2ed033293 P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal/wals/b9342dbe0497499ba4c238f2ed033293
I20240502 21:26:48.430598 31381 catalog_manager.cc:4830] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): tablet b9342dbe0497499ba4c238f2ed033293 (table rebalancer_extra_table_0 [id=9e962597ef7c41f493f1a41cd94c913b]) successfully deleted
I20240502 21:26:48.443938 1884 raft_consensus.cc:1050] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1: attempting to promote NON_VOTER d1406e92aec84ac585304e57bd37a40d to VOTER
I20240502 21:26:48.445036 1884 consensus_queue.cc:237] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } }
I20240502 21:26:48.448244 31795 raft_consensus.cc:1261] T 62f083e9641042c990efcfb1c4104de5 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:48.448355 31535 raft_consensus.cc:1261] T 62f083e9641042c990efcfb1c4104de5 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:48.448881 2777 consensus_queue.cc:1035] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:48.449188 2228 raft_consensus.cc:1261] T 62f083e9641042c990efcfb1c4104de5 P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:48.449476 2777 consensus_queue.cc:1035] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:48.451169 2777 consensus_queue.cc:1035] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:48.469691 2095 raft_consensus.cc:1203] T 83904952a5a541ecac508449e23e70dd P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:26:48.472170 31797 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b407c2c9b3a14ce3ada52b7856c6ecee"
dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49448
I20240502 21:26:48.472391 31797 raft_consensus.cc:595] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Received request to transfer leadership
I20240502 21:26:48.482511 1884 raft_consensus.cc:2932] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:48.483862 31795 raft_consensus.cc:2932] T 62f083e9641042c990efcfb1c4104de5 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:48.488992 31395 catalog_manager.cc:5484] T 62f083e9641042c990efcfb1c4104de5 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index 4 to 5, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:48.490044 2228 raft_consensus.cc:2932] T 62f083e9641042c990efcfb1c4104de5 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:48.492976 31535 raft_consensus.cc:2932] T 62f083e9641042c990efcfb1c4104de5 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:48.497459 3033 raft_consensus.cc:1050] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c: attempting to promote NON_VOTER d1406e92aec84ac585304e57bd37a40d to VOTER
I20240502 21:26:48.498517 3033 consensus_queue.cc:237] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } }
I20240502 21:26:48.501439 31925 consensus_queue.cc:237] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } }
I20240502 21:26:48.502933 2228 raft_consensus.cc:1261] T 62f083e9641042c990efcfb1c4104de5 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:48.503026 31795 raft_consensus.cc:1261] T 62f083e9641042c990efcfb1c4104de5 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:48.503572 2777 consensus_queue.cc:1035] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:48.503903 3086 consensus_queue.cc:1035] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:48.506994 3160 raft_consensus.cc:2932] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:48.507720 2228 raft_consensus.cc:2932] T 62f083e9641042c990efcfb1c4104de5 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:48.507927 31795 raft_consensus.cc:2932] T 62f083e9641042c990efcfb1c4104de5 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:48.522115 31395 catalog_manager.cc:5484] T 62f083e9641042c990efcfb1c4104de5 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index 5 to 6, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:48.531142 2974 raft_consensus.cc:482] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:48.531337 2974 raft_consensus.cc:504] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:48.532189 2974 leader_election.cc:288] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:48.532996 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5019de9e275c42a697d17fc31bceac36" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:48.533195 31382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 62f083e9641042c990efcfb1c4104de5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:48.533300 31535 raft_consensus.cc:2454] T 5019de9e275c42a697d17fc31bceac36 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:26:48.533773 1966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5019de9e275c42a697d17fc31bceac36" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:26:48.534021 1966 raft_consensus.cc:2454] T 5019de9e275c42a697d17fc31bceac36 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:26:48.534763 31603 leader_election.cc:302] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:48.535022 2228 raft_consensus.cc:1261] T 5b1dcae186f542cea92c9191fa8ef951 P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:48.535226 2974 raft_consensus.cc:2781] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:48.535389 2974 raft_consensus.cc:482] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:48.535481 2974 raft_consensus.cc:3037] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:48.537727 2799 consensus_queue.cc:1035] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:48.541967 2900 raft_consensus.cc:1050] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c: attempting to promote NON_VOTER 1e04157b98dd4fad98858c290b73c3db to VOTER
I20240502 21:26:48.542758 2900 consensus_queue.cc:237] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } }
I20240502 21:26:48.545121 3067 consensus_queue.cc:1035] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:48.549680 31535 raft_consensus.cc:1261] T 5b1dcae186f542cea92c9191fa8ef951 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:48.550452 3033 consensus_queue.cc:1035] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:48.551457 31516 tablet_service.cc:1514] Processing DeleteTablet for tablet 62f083e9641042c990efcfb1c4104de5 with delete_type TABLET_DATA_TOMBSTONED (TS a44f41b0b6644fb6aa29476aed8c964f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45194
I20240502 21:26:48.539628 2974 raft_consensus.cc:504] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:48.553694 2974 leader_election.cc:288] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Requested vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:48.553705 1966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5019de9e275c42a697d17fc31bceac36" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf"
I20240502 21:26:48.553777 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5019de9e275c42a697d17fc31bceac36" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:26:48.553894 1966 raft_consensus.cc:3037] T 5019de9e275c42a697d17fc31bceac36 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:48.554016 31535 raft_consensus.cc:3037] T 5019de9e275c42a697d17fc31bceac36 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:48.557317 1966 raft_consensus.cc:2454] T 5019de9e275c42a697d17fc31bceac36 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:48.557698 31535 raft_consensus.cc:2454] T 5019de9e275c42a697d17fc31bceac36 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:48.558236 31603 leader_election.cc:302] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:48.560068 31925 raft_consensus.cc:1261] T 5b1dcae186f542cea92c9191fa8ef951 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:48.561033 2799 consensus_queue.cc:1035] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:48.561496 31665 raft_consensus.cc:1261] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:48.562005 1833 raft_consensus.cc:1261] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:48.562980 2799 consensus_queue.cc:1035] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:48.563546 2799 consensus_queue.cc:1035] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:48.564862 31924 raft_consensus.cc:1261] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:48.565697 2791 consensus_queue.cc:1035] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:48.566679 3181 raft_consensus.cc:2781] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:48.573377 2791 raft_consensus.cc:2932] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:48.574416 3160 consensus_queue.cc:1035] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:48.578651 2228 raft_consensus.cc:2932] T 5b1dcae186f542cea92c9191fa8ef951 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:48.584980 31390 catalog_manager.cc:5484] T 5b1dcae186f542cea92c9191fa8ef951 P d1406e92aec84ac585304e57bd37a40d reported cstate change: config changed from index 4 to 5, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:48.589804 31924 raft_consensus.cc:2932] T 5b1dcae186f542cea92c9191fa8ef951 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:48.594316 3086 consensus_queue.cc:1035] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:48.594537 2799 raft_consensus.cc:2932] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:48.598934 2843 raft_consensus.cc:1050] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f: attempting to promote NON_VOTER 9dec645b473f40578e726f3e50e219bf to VOTER
I20240502 21:26:48.599970 2843 consensus_queue.cc:237] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:26:48.608162 31666 raft_consensus.cc:2932] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:48.609666 31925 raft_consensus.cc:2932] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:48.595002 3181 raft_consensus.cc:686] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Becoming Leader. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Running, Role: LEADER
I20240502 21:26:48.617426 31395 catalog_manager.cc:5484] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: config changed from index 4 to 5, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:48.623869 31795 consensus_queue.cc:237] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } }
I20240502 21:26:48.626014 2230 raft_consensus.cc:1261] T 5b1dcae186f542cea92c9191fa8ef951 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:48.626717 3033 consensus_queue.cc:1035] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:48.627513 3181 consensus_queue.cc:237] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:48.629870 2791 raft_consensus.cc:2932] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:48.633105 2230 raft_consensus.cc:2932] T 5b1dcae186f542cea92c9191fa8ef951 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:48.638706 31381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5b1dcae186f542cea92c9191fa8ef951 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:48.641113 31925 raft_consensus.cc:1261] T 5b1dcae186f542cea92c9191fa8ef951 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:48.642453 3033 consensus_queue.cc:1035] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:48.647028 31390 catalog_manager.cc:5484] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index 5 to 6, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:48.649715 31925 raft_consensus.cc:2932] T 5b1dcae186f542cea92c9191fa8ef951 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:48.656369 31389 catalog_manager.cc:5484] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130). New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:48.661788 3177 tablet_replica.cc:330] T 62f083e9641042c990efcfb1c4104de5 P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:48.662065 3177 raft_consensus.cc:2229] T 62f083e9641042c990efcfb1c4104de5 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:48.662366 31516 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b1dcae186f542cea92c9191fa8ef951 with delete_type TABLET_DATA_TOMBSTONED (TS a44f41b0b6644fb6aa29476aed8c964f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45194
I20240502 21:26:48.663431 1966 raft_consensus.cc:1261] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:48.663003 3177 raft_consensus.cc:2258] T 62f083e9641042c990efcfb1c4104de5 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:48.663712 31665 raft_consensus.cc:1261] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:48.664706 2631 consensus_queue.cc:1035] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:48.666865 2631 consensus_queue.cc:1035] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:48.672621 3177 ts_tablet_manager.cc:1875] T 62f083e9641042c990efcfb1c4104de5 P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:48.676846 31795 raft_consensus.cc:1261] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:48.677781 2631 consensus_queue.cc:1035] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:48.678833 3067 consensus_queue.cc:1035] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:48.682201 3177 ts_tablet_manager.cc:1888] T 62f083e9641042c990efcfb1c4104de5 P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:26:48.682313 3177 log.cc:1196] T 62f083e9641042c990efcfb1c4104de5 P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/62f083e9641042c990efcfb1c4104de5
I20240502 21:26:48.683068 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet 62f083e9641042c990efcfb1c4104de5 (table rebalance_test_table_17 [id=2e8e8260e3b64486b5712ad7289d1cc8]) successfully deleted
I20240502 21:26:48.686705 3033 raft_consensus.cc:979] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c: : Instructing follower 9dec645b473f40578e726f3e50e219bf to start an election
I20240502 21:26:48.686889 3033 raft_consensus.cc:1067] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Signalling peer 9dec645b473f40578e726f3e50e219bf to start an election
I20240502 21:26:48.687992 1965 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b407c2c9b3a14ce3ada52b7856c6ecee"
dest_uuid: "9dec645b473f40578e726f3e50e219bf"
from {username='slave'} at 127.24.226.131:41999
I20240502 21:26:48.688271 1965 raft_consensus.cc:482] T b407c2c9b3a14ce3ada52b7856c6ecee P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:26:48.688412 1965 raft_consensus.cc:3037] T b407c2c9b3a14ce3ada52b7856c6ecee P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:48.691448 1965 raft_consensus.cc:504] T b407c2c9b3a14ce3ada52b7856c6ecee P 9dec645b473f40578e726f3e50e219bf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:26:48.696318 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b407c2c9b3a14ce3ada52b7856c6ecee" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:48.696540 31797 raft_consensus.cc:3032] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:26:48.696640 31797 raft_consensus.cc:729] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Running, Role: LEADER
I20240502 21:26:48.696954 31797 consensus_queue.cc:260] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:26:48.697409 31797 raft_consensus.cc:3037] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:48.700366 31797 raft_consensus.cc:2454] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 2.
I20240502 21:26:48.701398 1965 leader_election.cc:288] T b407c2c9b3a14ce3ada52b7856c6ecee P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 2 election: Requested vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:48.700906 1834 raft_consensus.cc:2932] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:48.703620 31797 consensus_queue.cc:237] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } }
I20240502 21:26:48.696453 31535 raft_consensus.cc:2932] T 5b1dcae186f542cea92c9191fa8ef951 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:48.710175 31665 raft_consensus.cc:1261] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:48.710970 2791 consensus_queue.cc:1035] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:48.714453 2799 raft_consensus.cc:2932] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:48.718592 31381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9ea8ddb9f8e04893b56c10dbbb5af321 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:48.718791 2631 raft_consensus.cc:2932] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:48.720671 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b407c2c9b3a14ce3ada52b7856c6ecee" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:48.720942 31925 raft_consensus.cc:3037] T b407c2c9b3a14ce3ada52b7856c6ecee P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:48.721479 31389 catalog_manager.cc:5484] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index 5 to 6, VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) evicted. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:48.723538 31925 raft_consensus.cc:2454] T b407c2c9b3a14ce3ada52b7856c6ecee P 2d50f33557944bb480ea32199d9ee4f1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 2.
I20240502 21:26:48.724380 1902 leader_election.cc:302] T b407c2c9b3a14ce3ada52b7856c6ecee P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 2d50f33557944bb480ea32199d9ee4f1, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:26:48.726235 31905 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ea8ddb9f8e04893b56c10dbbb5af321 with delete_type TABLET_DATA_TOMBSTONED (TS 2d50f33557944bb480ea32199d9ee4f1 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47824
I20240502 21:26:48.729838 1833 raft_consensus.cc:1261] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:48.730604 2900 consensus_queue.cc:1035] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:48.732280 1834 raft_consensus.cc:2932] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:48.738745 3178 raft_consensus.cc:2781] T b407c2c9b3a14ce3ada52b7856c6ecee P 9dec645b473f40578e726f3e50e219bf [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:26:48.741170 31795 raft_consensus.cc:2932] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:48.749148 1966 raft_consensus.cc:2932] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:48.749584 31666 raft_consensus.cc:2932] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:48.756186 3178 raft_consensus.cc:686] T b407c2c9b3a14ce3ada52b7856c6ecee P 9dec645b473f40578e726f3e50e219bf [term 2 LEADER]: Becoming Leader. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Running, Role: LEADER
I20240502 21:26:48.756615 3178 consensus_queue.cc:237] T b407c2c9b3a14ce3ada52b7856c6ecee P 9dec645b473f40578e726f3e50e219bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:26:48.760339 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b407c2c9b3a14ce3ada52b7856c6ecee" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:48.760545 31665 raft_consensus.cc:3037] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:48.760555 31389 catalog_manager.cc:5484] T b407c2c9b3a14ce3ada52b7856c6ecee P 9dec645b473f40578e726f3e50e219bf reported cstate change: term changed from 1 to 2, leader changed from 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) to 9dec645b473f40578e726f3e50e219bf (127.24.226.134). New cstate: current_term: 2 leader_uuid: "9dec645b473f40578e726f3e50e219bf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:48.761919 31667 raft_consensus.cc:2932] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:48.763934 31665 raft_consensus.cc:2454] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 2.
I20240502 21:26:48.766515 31389 catalog_manager.cc:5484] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 9dec645b473f40578e726f3e50e219bf reported cstate change: config changed from index 4 to 5, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:48.770809 3177 tablet_replica.cc:330] T 5b1dcae186f542cea92c9191fa8ef951 P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:48.771054 3177 raft_consensus.cc:2229] T 5b1dcae186f542cea92c9191fa8ef951 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:48.774116 2891 consensus_queue.cc:1035] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:48.783874 3177 raft_consensus.cc:2258] T 5b1dcae186f542cea92c9191fa8ef951 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:48.784885 3177 ts_tablet_manager.cc:1875] T 5b1dcae186f542cea92c9191fa8ef951 P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:48.788103 3206 tablet_replica.cc:330] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:26:48.789417 3206 raft_consensus.cc:2229] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:48.790877 3206 raft_consensus.cc:2258] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:48.792831 3177 ts_tablet_manager.cc:1888] T 5b1dcae186f542cea92c9191fa8ef951 P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:26:48.793015 3177 log.cc:1196] T 5b1dcae186f542cea92c9191fa8ef951 P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/5b1dcae186f542cea92c9191fa8ef951
I20240502 21:26:48.796955 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet 5b1dcae186f542cea92c9191fa8ef951 (table rebalance_test_table_13 [id=c7511c25586c4ba086fcfffa353da437]) successfully deleted
I20240502 21:26:48.799754 3206 ts_tablet_manager.cc:1875] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:48.807045 2891 consensus_queue.cc:1035] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:48.807862 3206 ts_tablet_manager.cc:1888] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:26:48.808002 3206 log.cc:1196] T 9ea8ddb9f8e04893b56c10dbbb5af321 P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/9ea8ddb9f8e04893b56c10dbbb5af321
I20240502 21:26:48.809188 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet 9ea8ddb9f8e04893b56c10dbbb5af321 (table rebalance_test_table_11 [id=c46054f8169d4487b3c912c0681b1c3a]) successfully deleted
I20240502 21:26:48.816583 31535 consensus_queue.cc:237] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:26:48.819808 31666 raft_consensus.cc:1261] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:48.820323 1966 raft_consensus.cc:1261] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:48.820469 3204 consensus_queue.cc:1035] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:48.820896 3205 consensus_queue.cc:1035] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:48.823217 3207 raft_consensus.cc:2932] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:48.824609 1966 raft_consensus.cc:2932] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:48.825414 31666 raft_consensus.cc:2932] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:48.832844 31382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 55fc18e4cc98446ca50c1dd2b6a9a89d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:48.835353 31389 catalog_manager.cc:5484] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: config changed from index 5 to 6, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:48.838863 31775 tablet_service.cc:1514] Processing DeleteTablet for tablet 55fc18e4cc98446ca50c1dd2b6a9a89d with delete_type TABLET_DATA_TOMBSTONED (TS 08d7a0680d244b55b93d17f9ab57b84c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51596
I20240502 21:26:48.840178 31395 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_extra_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:48.842557 3219 tablet_replica.cc:330] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 08d7a0680d244b55b93d17f9ab57b84c: stopping tablet replica
I20240502 21:26:48.842861 3219 raft_consensus.cc:2229] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:48.843202 3219 raft_consensus.cc:2258] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:48.844257 3219 ts_tablet_manager.cc:1875] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 08d7a0680d244b55b93d17f9ab57b84c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:48.850211 31647 tablet_service.cc:1467] Processing CreateTablet for tablet de2216a74f8344779a881651b66a85e5 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=daf389fa76d84011ad57fe7156cfa6a8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:48.850654 31647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de2216a74f8344779a881651b66a85e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:48.853224 1812 tablet_service.cc:1467] Processing CreateTablet for tablet de2216a74f8344779a881651b66a85e5 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=daf389fa76d84011ad57fe7156cfa6a8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:48.853754 1812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de2216a74f8344779a881651b66a85e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:48.854192 2077 tablet_service.cc:1467] Processing CreateTablet for tablet de2216a74f8344779a881651b66a85e5 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=daf389fa76d84011ad57fe7156cfa6a8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:48.854686 2077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de2216a74f8344779a881651b66a85e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:48.854686 1946 tablet_service.cc:1467] Processing CreateTablet for tablet f13c3b9c4b7c471fa34156b781473ec4 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=daf389fa76d84011ad57fe7156cfa6a8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:48.855089 1946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f13c3b9c4b7c471fa34156b781473ec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:48.856791 1811 tablet_service.cc:1467] Processing CreateTablet for tablet f13c3b9c4b7c471fa34156b781473ec4 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=daf389fa76d84011ad57fe7156cfa6a8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:48.857170 1811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f13c3b9c4b7c471fa34156b781473ec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:48.857885 2075 tablet_service.cc:1467] Processing CreateTablet for tablet 59f36d947f4341f39b6969cc1e065b6c (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=daf389fa76d84011ad57fe7156cfa6a8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:48.858075 2210 tablet_service.cc:1467] Processing CreateTablet for tablet 59f36d947f4341f39b6969cc1e065b6c (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=daf389fa76d84011ad57fe7156cfa6a8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:48.858248 2075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59f36d947f4341f39b6969cc1e065b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:48.858474 2210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59f36d947f4341f39b6969cc1e065b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:48.861255 31905 tablet_service.cc:1467] Processing CreateTablet for tablet 59f36d947f4341f39b6969cc1e065b6c (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=daf389fa76d84011ad57fe7156cfa6a8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:48.861732 2208 tablet_service.cc:1467] Processing CreateTablet for tablet f13c3b9c4b7c471fa34156b781473ec4 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=daf389fa76d84011ad57fe7156cfa6a8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:48.861804 31905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59f36d947f4341f39b6969cc1e065b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:48.862068 2208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f13c3b9c4b7c471fa34156b781473ec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:48.878749 31395 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:26:48.899619 3222 tablet_bootstrap.cc:492] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:48.901206 2791 raft_consensus.cc:1050] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c: attempting to promote NON_VOTER 9dec645b473f40578e726f3e50e219bf to VOTER
I20240502 21:26:48.901723 3221 tablet_bootstrap.cc:492] T 59f36d947f4341f39b6969cc1e065b6c P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:48.902009 2791 consensus_queue.cc:237] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:26:48.902724 3222 tablet_bootstrap.cc:654] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:48.904067 3221 tablet_bootstrap.cc:654] T 59f36d947f4341f39b6969cc1e065b6c P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:48.906499 3223 tablet_bootstrap.cc:492] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:48.908383 3223 tablet_bootstrap.cc:654] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:48.915897 3220 tablet_bootstrap.cc:492] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:48.918423 3219 ts_tablet_manager.cc:1888] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 08d7a0680d244b55b93d17f9ab57b84c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:26:48.918707 3219 log.cc:1196] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 08d7a0680d244b55b93d17f9ab57b84c: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/wals/55fc18e4cc98446ca50c1dd2b6a9a89d
I20240502 21:26:48.919572 3223 tablet_bootstrap.cc:492] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:48.919719 31381 catalog_manager.cc:4830] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): tablet 55fc18e4cc98446ca50c1dd2b6a9a89d (table rebalance_test_table_11 [id=c46054f8169d4487b3c912c0681b1c3a]) successfully deleted
I20240502 21:26:48.921176 3223 ts_tablet_manager.cc:1367] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240502 21:26:48.922003 3221 tablet_bootstrap.cc:492] T 59f36d947f4341f39b6969cc1e065b6c P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:48.922700 3232 tablet_bootstrap.cc:492] T 59f36d947f4341f39b6969cc1e065b6c P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:48.922775 3223 raft_consensus.cc:348] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:48.924664 3232 tablet_bootstrap.cc:654] T 59f36d947f4341f39b6969cc1e065b6c P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:48.924723 3220 tablet_bootstrap.cc:654] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:48.925653 3221 ts_tablet_manager.cc:1367] T 59f36d947f4341f39b6969cc1e065b6c P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.003s
I20240502 21:26:48.927310 3222 tablet_bootstrap.cc:492] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:48.927495 3222 ts_tablet_manager.cc:1367] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.002s
I20240502 21:26:48.927677 1966 raft_consensus.cc:1261] T f9bbc56ba2f841b9bb36a864e1a1f808 P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:48.927991 31925 raft_consensus.cc:1261] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:48.928232 31535 raft_consensus.cc:1261] T 5019de9e275c42a697d17fc31bceac36 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:48.928287 3222 raft_consensus.cc:348] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:48.928622 2799 consensus_queue.cc:1035] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:48.928885 3222 raft_consensus.cc:374] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:48.928892 3182 consensus_queue.cc:1035] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:48.929097 3222 raft_consensus.cc:729] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:48.929132 2900 consensus_queue.cc:1035] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:48.929667 3221 raft_consensus.cc:348] T 59f36d947f4341f39b6969cc1e065b6c P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:48.930193 31666 raft_consensus.cc:1261] T f9bbc56ba2f841b9bb36a864e1a1f808 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:48.930379 3222 consensus_queue.cc:260] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:48.930691 3223 raft_consensus.cc:374] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:48.930859 3223 raft_consensus.cc:729] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:48.931248 3033 consensus_queue.cc:1035] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:48.931248 3223 consensus_queue.cc:260] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:48.932579 3222 ts_tablet_manager.cc:1398] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:26:48.929862 3221 raft_consensus.cc:374] T 59f36d947f4341f39b6969cc1e065b6c P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:48.932773 3221 raft_consensus.cc:729] T 59f36d947f4341f39b6969cc1e065b6c P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:48.937793 3223 ts_tablet_manager.cc:1398] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240502 21:26:48.937913 1965 raft_consensus.cc:1261] T 5019de9e275c42a697d17fc31bceac36 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:48.938436 3182 consensus_queue.cc:1035] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:48.938899 3222 tablet_bootstrap.cc:492] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:48.939889 3224 tablet_bootstrap.cc:492] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:48.939062 3221 consensus_queue.cc:260] T 59f36d947f4341f39b6969cc1e065b6c P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:48.942819 3224 tablet_bootstrap.cc:654] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:48.943468 3223 tablet_bootstrap.cc:492] T de2216a74f8344779a881651b66a85e5 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:48.944612 3221 ts_tablet_manager.cc:1398] T 59f36d947f4341f39b6969cc1e065b6c P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.019s user 0.000s sys 0.002s
I20240502 21:26:48.948372 3223 tablet_bootstrap.cc:654] T de2216a74f8344779a881651b66a85e5 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:48.948596 2429 raft_consensus.cc:482] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:48.948762 2429 raft_consensus.cc:504] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:48.949520 2429 leader_election.cc:288] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:26:48.950819 1966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13c3b9c4b7c471fa34156b781473ec4" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:26:48.948498 3221 tablet_bootstrap.cc:492] T de2216a74f8344779a881651b66a85e5 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
W20240502 21:26:48.951517 2161 leader_election.cc:341] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:48.952556 3222 tablet_bootstrap.cc:654] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:48.953490 3232 tablet_bootstrap.cc:492] T 59f36d947f4341f39b6969cc1e065b6c P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:48.953709 3232 ts_tablet_manager.cc:1367] T 59f36d947f4341f39b6969cc1e065b6c P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.006s
I20240502 21:26:48.954728 3232 raft_consensus.cc:348] T 59f36d947f4341f39b6969cc1e065b6c P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:48.955013 3232 raft_consensus.cc:374] T 59f36d947f4341f39b6969cc1e065b6c P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:48.955113 3232 raft_consensus.cc:729] T 59f36d947f4341f39b6969cc1e065b6c P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:48.955430 3232 consensus_queue.cc:260] T 59f36d947f4341f39b6969cc1e065b6c P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:48.956172 3232 ts_tablet_manager.cc:1398] T 59f36d947f4341f39b6969cc1e065b6c P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:48.956692 31390 catalog_manager.cc:5576] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039) does not have the latest schema for tablet 5019de9e275c42a697d17fc31bceac36 (table rebalancer_dynamic_table [id=df46a59a7eee43beba75e74228f448ac]). Expected version 1 got 0
I20240502 21:26:48.950423 1833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13c3b9c4b7c471fa34156b781473ec4" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:26:48.957053 1833 raft_consensus.cc:2454] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:26:48.957494 2163 leader_election.cc:302] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, d1406e92aec84ac585304e57bd37a40d; no voters: 9dec645b473f40578e726f3e50e219bf
I20240502 21:26:48.958470 2429 raft_consensus.cc:2781] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:48.958564 2429 raft_consensus.cc:482] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:48.958663 2429 raft_consensus.cc:3037] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:48.960083 3224 tablet_bootstrap.cc:492] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:48.960835 3224 ts_tablet_manager.cc:1367] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.005s
I20240502 21:26:48.961866 2429 raft_consensus.cc:504] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:48.962450 3224 raft_consensus.cc:348] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:48.963130 1833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13c3b9c4b7c471fa34156b781473ec4" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:26:48.963333 1833 raft_consensus.cc:3037] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:48.964170 3224 raft_consensus.cc:374] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:48.964308 3224 raft_consensus.cc:729] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:48.964926 3244 tablet.cc:1722] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:48.965507 3244 tablet.cc:1620] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada: MemRowSet was empty: no flush needed.
I20240502 21:26:48.966274 1833 raft_consensus.cc:2454] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 1.
I20240502 21:26:48.966964 2163 leader_election.cc:302] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:48.967334 3213 raft_consensus.cc:2781] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:48.967499 3213 raft_consensus.cc:686] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Becoming Leader. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Running, Role: LEADER
I20240502 21:26:48.967820 3213 consensus_queue.cc:237] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:48.969822 3224 consensus_queue.cc:260] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:48.971056 3224 ts_tablet_manager.cc:1398] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240502 21:26:48.971820 31389 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=df46a59a7eee43beba75e74228f448ac] alter complete (version 1)
I20240502 21:26:48.973562 3245 tablet.cc:1722] T 5019de9e275c42a697d17fc31bceac36 P a44f41b0b6644fb6aa29476aed8c964f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:48.975584 2429 leader_election.cc:288] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Requested vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:26:48.976109 3245 tablet.cc:1620] T 5019de9e275c42a697d17fc31bceac36 P a44f41b0b6644fb6aa29476aed8c964f: MemRowSet was empty: no flush needed.
I20240502 21:26:48.976612 31389 catalog_manager.cc:5576] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039) does not have the latest schema for tablet 5019de9e275c42a697d17fc31bceac36 (table rebalancer_dynamic_table [id=df46a59a7eee43beba75e74228f448ac]). Expected version 1 got 0
I20240502 21:26:48.978087 1966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13c3b9c4b7c471fa34156b781473ec4" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf"
W20240502 21:26:48.978523 2161 leader_election.cc:341] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:48.984344 3220 tablet_bootstrap.cc:492] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:26:48.984675 3220 ts_tablet_manager.cc:1367] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.069s user 0.004s sys 0.000s
I20240502 21:26:48.985581 3220 raft_consensus.cc:348] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:48.986600 3220 raft_consensus.cc:374] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:48.986817 3220 raft_consensus.cc:729] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:26:48.987218 3220 consensus_queue.cc:260] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:48.989054 3220 ts_tablet_manager.cc:1398] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 21:26:48.991180 3221 tablet_bootstrap.cc:654] T de2216a74f8344779a881651b66a85e5 P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:48.992537 3033 raft_consensus.cc:2932] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:48.994066 31390 catalog_manager.cc:5484] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d reported cstate change: term changed from 0 to 1, leader changed from <none> to d1406e92aec84ac585304e57bd37a40d (127.24.226.136). New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:48.997215 3222 tablet_bootstrap.cc:492] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:48.997778 3222 ts_tablet_manager.cc:1367] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.000s
I20240502 21:26:48.998687 3222 raft_consensus.cc:348] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:48.998462 31925 raft_consensus.cc:2932] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:48.998988 3222 raft_consensus.cc:374] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:48.999897 3222 raft_consensus.cc:729] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:49.000320 3222 consensus_queue.cc:260] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:49.001632 3222 ts_tablet_manager.cc:1398] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 21:26:49.004112 1966 raft_consensus.cc:2932] T f9bbc56ba2f841b9bb36a864e1a1f808 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:49.004554 31389 catalog_manager.cc:5484] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: config changed from index 4 to 5, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:49.007495 3213 raft_consensus.cc:482] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:49.007712 3213 raft_consensus.cc:504] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:49.010198 2095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59f36d947f4341f39b6969cc1e065b6c" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:49.010413 2095 raft_consensus.cc:2454] T 59f36d947f4341f39b6969cc1e065b6c P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:26:49.009794 3213 leader_election.cc:288] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:49.010915 2163 leader_election.cc:302] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:49.011382 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59f36d947f4341f39b6969cc1e065b6c" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:49.011588 31925 raft_consensus.cc:2454] T 59f36d947f4341f39b6969cc1e065b6c P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:26:49.016881 3213 raft_consensus.cc:2781] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:49.016968 3213 raft_consensus.cc:482] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:49.017072 3213 raft_consensus.cc:3037] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:49.020071 3213 raft_consensus.cc:504] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:49.021019 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59f36d947f4341f39b6969cc1e065b6c" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:49.021121 3213 leader_election.cc:288] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Requested vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:49.021248 31925 raft_consensus.cc:3037] T 59f36d947f4341f39b6969cc1e065b6c P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:49.021728 2095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59f36d947f4341f39b6969cc1e065b6c" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:26:49.021941 2095 raft_consensus.cc:3037] T 59f36d947f4341f39b6969cc1e065b6c P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:49.021509 31666 raft_consensus.cc:2932] T f9bbc56ba2f841b9bb36a864e1a1f808 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:49.024214 31925 raft_consensus.cc:2454] T 59f36d947f4341f39b6969cc1e065b6c P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 1.
I20240502 21:26:49.024653 2164 leader_election.cc:302] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:49.025133 2095 raft_consensus.cc:2454] T 59f36d947f4341f39b6969cc1e065b6c P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 1.
I20240502 21:26:49.025441 3213 raft_consensus.cc:2781] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:49.027587 3213 raft_consensus.cc:686] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Becoming Leader. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Running, Role: LEADER
I20240502 21:26:49.027870 3213 consensus_queue.cc:237] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:49.029470 31389 catalog_manager.cc:5576] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753) does not have the latest schema for tablet 5019de9e275c42a697d17fc31bceac36 (table rebalancer_dynamic_table [id=df46a59a7eee43beba75e74228f448ac]). Expected version 1 got 0
I20240502 21:26:49.030117 3223 tablet_bootstrap.cc:492] T de2216a74f8344779a881651b66a85e5 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:49.030333 3223 ts_tablet_manager.cc:1367] T de2216a74f8344779a881651b66a85e5 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.087s user 0.010s sys 0.000s
I20240502 21:26:49.032006 3223 raft_consensus.cc:348] T de2216a74f8344779a881651b66a85e5 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:49.032337 3223 raft_consensus.cc:374] T de2216a74f8344779a881651b66a85e5 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:49.032480 3223 raft_consensus.cc:729] T de2216a74f8344779a881651b66a85e5 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:49.032744 3221 tablet_bootstrap.cc:492] T de2216a74f8344779a881651b66a85e5 P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:49.032822 3223 consensus_queue.cc:260] T de2216a74f8344779a881651b66a85e5 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:49.032967 3221 ts_tablet_manager.cc:1367] T de2216a74f8344779a881651b66a85e5 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.087s user 0.003s sys 0.009s
I20240502 21:26:49.033738 3221 raft_consensus.cc:348] T de2216a74f8344779a881651b66a85e5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:49.034044 3221 raft_consensus.cc:374] T de2216a74f8344779a881651b66a85e5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:49.034183 3221 raft_consensus.cc:729] T de2216a74f8344779a881651b66a85e5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:49.034505 3221 consensus_queue.cc:260] T de2216a74f8344779a881651b66a85e5 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:49.035127 3221 ts_tablet_manager.cc:1398] T de2216a74f8344779a881651b66a85e5 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:49.036536 3223 ts_tablet_manager.cc:1398] T de2216a74f8344779a881651b66a85e5 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 21:26:49.039582 3251 tablet.cc:1722] T 5019de9e275c42a697d17fc31bceac36 P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:49.042429 3251 tablet.cc:1620] T 5019de9e275c42a697d17fc31bceac36 P 9dec645b473f40578e726f3e50e219bf: MemRowSet was empty: no flush needed.
I20240502 21:26:49.043915 31389 catalog_manager.cc:5576] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753) does not have the latest schema for tablet 5019de9e275c42a697d17fc31bceac36 (table rebalancer_dynamic_table [id=df46a59a7eee43beba75e74228f448ac]). Expected version 1 got 0
I20240502 21:26:49.047792 3086 raft_consensus.cc:1050] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1: attempting to promote NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b to VOTER
I20240502 21:26:49.048828 3086 consensus_queue.cc:237] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } }
I20240502 21:26:49.055743 31795 raft_consensus.cc:1261] T 83904952a5a541ecac508449e23e70dd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:49.055824 31535 raft_consensus.cc:1261] T 83904952a5a541ecac508449e23e70dd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:49.056470 3160 consensus_queue.cc:1035] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:49.056692 2777 consensus_queue.cc:1035] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:49.057607 2095 raft_consensus.cc:1261] T 83904952a5a541ecac508449e23e70dd P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:49.058348 3160 consensus_queue.cc:1035] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:49.060674 31389 catalog_manager.cc:5484] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d reported cstate change: term changed from 0 to 1, leader changed from <none> to d1406e92aec84ac585304e57bd37a40d (127.24.226.136). New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:49.067198 3160 raft_consensus.cc:2932] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:49.069164 31795 raft_consensus.cc:2932] T 83904952a5a541ecac508449e23e70dd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:49.069878 31535 raft_consensus.cc:2932] T 83904952a5a541ecac508449e23e70dd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:49.074889 31389 catalog_manager.cc:5484] T 83904952a5a541ecac508449e23e70dd P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:49.076272 2095 raft_consensus.cc:2932] T 83904952a5a541ecac508449e23e70dd P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:49.079855 31395 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:26:49.082507 31925 consensus_queue.cc:237] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } }
I20240502 21:26:49.088748 31797 raft_consensus.cc:1261] T 83904952a5a541ecac508449e23e70dd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:49.088863 2095 raft_consensus.cc:1261] T 83904952a5a541ecac508449e23e70dd P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:49.089371 1884 consensus_queue.cc:1035] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:49.089684 2777 consensus_queue.cc:1035] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:49.091974 1884 raft_consensus.cc:2932] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:49.092701 31795 raft_consensus.cc:2932] T 83904952a5a541ecac508449e23e70dd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:49.095765 3244 tablet.cc:1722] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:49.096246 3244 tablet.cc:1620] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada: MemRowSet was empty: no flush needed.
I20240502 21:26:49.096621 31382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 83904952a5a541ecac508449e23e70dd with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:49.098011 31395 catalog_manager.cc:5484] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: config changed from index 5 to 6, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:49.098572 3251 tablet.cc:1722] T 5019de9e275c42a697d17fc31bceac36 P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:49.099033 3251 tablet.cc:1620] T 5019de9e275c42a697d17fc31bceac36 P 9dec645b473f40578e726f3e50e219bf: MemRowSet was empty: no flush needed.
I20240502 21:26:49.104828 31516 tablet_service.cc:1514] Processing DeleteTablet for tablet 83904952a5a541ecac508449e23e70dd with delete_type TABLET_DATA_TOMBSTONED (TS a44f41b0b6644fb6aa29476aed8c964f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45194
I20240502 21:26:49.105136 3177 tablet_replica.cc:330] T 83904952a5a541ecac508449e23e70dd P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:49.105316 3177 raft_consensus.cc:2229] T 83904952a5a541ecac508449e23e70dd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:49.105569 3177 raft_consensus.cc:2258] T 83904952a5a541ecac508449e23e70dd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:49.106143 3245 tablet.cc:1722] T 5019de9e275c42a697d17fc31bceac36 P a44f41b0b6644fb6aa29476aed8c964f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:49.106642 3245 tablet.cc:1620] T 5019de9e275c42a697d17fc31bceac36 P a44f41b0b6644fb6aa29476aed8c964f: MemRowSet was empty: no flush needed.
I20240502 21:26:49.107002 3177 ts_tablet_manager.cc:1875] T 83904952a5a541ecac508449e23e70dd P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:49.107261 31389 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=df46a59a7eee43beba75e74228f448ac] alter complete (version 2)
I20240502 21:26:49.114521 3177 ts_tablet_manager.cc:1888] T 83904952a5a541ecac508449e23e70dd P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:26:49.115116 2095 raft_consensus.cc:2932] T 83904952a5a541ecac508449e23e70dd P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:49.116739 3177 log.cc:1196] T 83904952a5a541ecac508449e23e70dd P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/83904952a5a541ecac508449e23e70dd
I20240502 21:26:49.117615 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet 83904952a5a541ecac508449e23e70dd (table rebalance_test_table_15 [id=7ff12e5a4c674a5fb7e07cb593390fd1]) successfully deleted
I20240502 21:26:49.118851 3178 raft_consensus.cc:482] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:49.119053 3178 raft_consensus.cc:504] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:49.119931 3178 leader_election.cc:288] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:26:49.120810 2228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13c3b9c4b7c471fa34156b781473ec4" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:26:49.120854 1833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13c3b9c4b7c471fa34156b781473ec4" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:26:49.121107 1833 raft_consensus.cc:2379] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9dec645b473f40578e726f3e50e219bf in current term 1: Already voted for candidate d1406e92aec84ac585304e57bd37a40d in this term.
I20240502 21:26:49.121697 1901 leader_election.cc:302] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dec645b473f40578e726f3e50e219bf; no voters: 1e04157b98dd4fad98858c290b73c3db, d1406e92aec84ac585304e57bd37a40d
I20240502 21:26:49.122146 3178 raft_consensus.cc:3037] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:49.125733 3178 raft_consensus.cc:2726] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:26:49.141716 31795 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f9bbc56ba2f841b9bb36a864e1a1f808"
dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49496
I20240502 21:26:49.141901 31795 raft_consensus.cc:595] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Received request to transfer leadership
I20240502 21:26:49.201098 31666 raft_consensus.cc:1261] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada [term 2 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:26:49.201941 3178 consensus_queue.cc:1035] T b407c2c9b3a14ce3ada52b7856c6ecee P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:49.207175 31795 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f9bbc56ba2f841b9bb36a864e1a1f808"
dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49510
I20240502 21:26:49.207360 31795 raft_consensus.cc:595] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Received request to transfer leadership
I20240502 21:26:49.217648 31389 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:26:49.226583 3244 tablet.cc:1722] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:49.227955 3251 tablet.cc:1722] T 5019de9e275c42a697d17fc31bceac36 P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:49.229720 3245 tablet.cc:1722] T 5019de9e275c42a697d17fc31bceac36 P a44f41b0b6644fb6aa29476aed8c964f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:49.236171 31395 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table_new_name [id=df46a59a7eee43beba75e74228f448ac] alter complete (version 3)
I20240502 21:26:49.242556 31795 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f9bbc56ba2f841b9bb36a864e1a1f808"
dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49536
I20240502 21:26:49.242779 31795 raft_consensus.cc:595] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Received request to transfer leadership
I20240502 21:26:49.265744 31925 raft_consensus.cc:1261] T b407c2c9b3a14ce3ada52b7856c6ecee P 2d50f33557944bb480ea32199d9ee4f1 [term 2 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:26:49.266201 3178 consensus_queue.cc:1035] T b407c2c9b3a14ce3ada52b7856c6ecee P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:49.269743 31795 raft_consensus.cc:1261] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [term 2 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:26:49.270471 3326 consensus_queue.cc:1035] T b407c2c9b3a14ce3ada52b7856c6ecee P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:49.272886 1965 consensus_queue.cc:237] T b407c2c9b3a14ce3ada52b7856c6ecee P 9dec645b473f40578e726f3e50e219bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:26:49.275090 31925 raft_consensus.cc:1261] T b407c2c9b3a14ce3ada52b7856c6ecee P 2d50f33557944bb480ea32199d9ee4f1 [term 2 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 21:26:49.275610 3327 consensus_queue.cc:1035] T b407c2c9b3a14ce3ada52b7856c6ecee P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:26:49.276298 31666 raft_consensus.cc:1261] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada [term 2 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 21:26:49.276849 3327 consensus_queue.cc:1035] T b407c2c9b3a14ce3ada52b7856c6ecee P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:26:49.277787 3327 raft_consensus.cc:2932] T b407c2c9b3a14ce3ada52b7856c6ecee P 9dec645b473f40578e726f3e50e219bf [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:49.281040 31925 raft_consensus.cc:2932] T b407c2c9b3a14ce3ada52b7856c6ecee P 2d50f33557944bb480ea32199d9ee4f1 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:49.281664 31666 raft_consensus.cc:2932] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:49.283053 31389 catalog_manager.cc:5484] T b407c2c9b3a14ce3ada52b7856c6ecee P 9dec645b473f40578e726f3e50e219bf reported cstate change: config changed from index 5 to 7, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New cstate: current_term: 2 leader_uuid: "9dec645b473f40578e726f3e50e219bf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:49.285282 31379 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b407c2c9b3a14ce3ada52b7856c6ecee with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:49.287829 31775 tablet_service.cc:1514] Processing DeleteTablet for tablet b407c2c9b3a14ce3ada52b7856c6ecee with delete_type TABLET_DATA_TOMBSTONED (TS 08d7a0680d244b55b93d17f9ab57b84c not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51596
I20240502 21:26:49.288293 3219 tablet_replica.cc:330] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c: stopping tablet replica
I20240502 21:26:49.288547 3219 raft_consensus.cc:2229] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:49.288729 3219 raft_consensus.cc:2258] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:49.289646 3219 ts_tablet_manager.cc:1875] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:49.291124 2974 raft_consensus.cc:482] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:49.291265 2974 raft_consensus.cc:504] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:49.291862 2974 leader_election.cc:288] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:26:49.292466 2095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de2216a74f8344779a881651b66a85e5" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:49.292677 2095 raft_consensus.cc:2454] T de2216a74f8344779a881651b66a85e5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:26:49.292627 1833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de2216a74f8344779a881651b66a85e5" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:26:49.292876 1833 raft_consensus.cc:2454] T de2216a74f8344779a881651b66a85e5 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:26:49.293064 31602 leader_election.cc:302] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 1ade0354c44c4c37ae480c7a94b11e1b; no voters:
I20240502 21:26:49.293752 2974 raft_consensus.cc:2781] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:49.293869 2974 raft_consensus.cc:482] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:49.293951 2974 raft_consensus.cc:3037] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:49.296228 3219 ts_tablet_manager.cc:1888] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 21:26:49.296358 3219 log.cc:1196] T b407c2c9b3a14ce3ada52b7856c6ecee P 08d7a0680d244b55b93d17f9ab57b84c: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/wals/b407c2c9b3a14ce3ada52b7856c6ecee
I20240502 21:26:49.296458 2974 raft_consensus.cc:504] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:49.297113 31381 catalog_manager.cc:4830] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): tablet b407c2c9b3a14ce3ada52b7856c6ecee (table rebalance_test_table_1 [id=08ebfd17dee84efd8a264a88ead3fa46]) successfully deleted
I20240502 21:26:49.297407 2095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de2216a74f8344779a881651b66a85e5" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:26:49.297585 2095 raft_consensus.cc:3037] T de2216a74f8344779a881651b66a85e5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:49.299269 2095 raft_consensus.cc:2454] T de2216a74f8344779a881651b66a85e5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:49.299638 31602 leader_election.cc:302] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 1ade0354c44c4c37ae480c7a94b11e1b; no voters:
I20240502 21:26:49.299634 2974 leader_election.cc:288] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Requested vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:26:49.299971 2974 raft_consensus.cc:2781] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:49.300112 2974 raft_consensus.cc:686] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Becoming Leader. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Running, Role: LEADER
I20240502 21:26:49.300401 1833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de2216a74f8344779a881651b66a85e5" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:26:49.300383 2974 consensus_queue.cc:237] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:49.300628 1833 raft_consensus.cc:3037] T de2216a74f8344779a881651b66a85e5 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:49.303032 1833 raft_consensus.cc:2454] T de2216a74f8344779a881651b66a85e5 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:49.304090 31389 catalog_manager.cc:5484] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130). New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:49.338941 31389 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:49.339149 31389 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:49.341586 31389 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 5019de9e275c42a697d17fc31bceac36
I20240502 21:26:49.342244 31647 tablet_service.cc:1514] Processing DeleteTablet for tablet 5019de9e275c42a697d17fc31bceac36 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:49 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:49.342314 1946 tablet_service.cc:1514] Processing DeleteTablet for tablet 5019de9e275c42a697d17fc31bceac36 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:49 UTC) from {username='slave'} at 127.0.0.1:33002
I20240502 21:26:49.342739 31516 tablet_service.cc:1514] Processing DeleteTablet for tablet 5019de9e275c42a697d17fc31bceac36 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:49 UTC) from {username='slave'} at 127.0.0.1:45194
I20240502 21:26:49.342916 3331 tablet_replica.cc:330] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:49.342984 3177 tablet_replica.cc:330] T 5019de9e275c42a697d17fc31bceac36 P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:49.343132 3331 raft_consensus.cc:2229] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:49.343243 3177 raft_consensus.cc:2229] T 5019de9e275c42a697d17fc31bceac36 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:49.343430 3331 raft_consensus.cc:2258] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:49.343497 3177 raft_consensus.cc:2258] T 5019de9e275c42a697d17fc31bceac36 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:49.344143 3331 ts_tablet_manager.cc:1875] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:49.344647 3177 ts_tablet_manager.cc:1875] T 5019de9e275c42a697d17fc31bceac36 P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:49.350975 3331 ts_tablet_manager.cc:1888] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:49.351133 3331 log.cc:1196] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/5019de9e275c42a697d17fc31bceac36
I20240502 21:26:49.351572 3331 ts_tablet_manager.cc:1909] T 5019de9e275c42a697d17fc31bceac36 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:26:49.352552 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet 5019de9e275c42a697d17fc31bceac36 (table rebalancer_dynamic_table_new_name [id=df46a59a7eee43beba75e74228f448ac]) successfully deleted
I20240502 21:26:49.354056 3177 ts_tablet_manager.cc:1888] T 5019de9e275c42a697d17fc31bceac36 P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:49.354485 3177 log.cc:1196] T 5019de9e275c42a697d17fc31bceac36 P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/5019de9e275c42a697d17fc31bceac36
I20240502 21:26:49.354945 3177 ts_tablet_manager.cc:1909] T 5019de9e275c42a697d17fc31bceac36 P a44f41b0b6644fb6aa29476aed8c964f: Deleting consensus metadata
I20240502 21:26:49.356092 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet 5019de9e275c42a697d17fc31bceac36 (table rebalancer_dynamic_table_new_name [id=df46a59a7eee43beba75e74228f448ac]) successfully deleted
I20240502 21:26:49.356349 3332 tablet_replica.cc:330] T 5019de9e275c42a697d17fc31bceac36 P 9dec645b473f40578e726f3e50e219bf: stopping tablet replica
I20240502 21:26:49.356560 3332 raft_consensus.cc:2229] T 5019de9e275c42a697d17fc31bceac36 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:49.356735 3332 raft_consensus.cc:2258] T 5019de9e275c42a697d17fc31bceac36 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:49.357372 3332 ts_tablet_manager.cc:1875] T 5019de9e275c42a697d17fc31bceac36 P 9dec645b473f40578e726f3e50e219bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:49.361716 3332 ts_tablet_manager.cc:1888] T 5019de9e275c42a697d17fc31bceac36 P 9dec645b473f40578e726f3e50e219bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:49.361809 3332 log.cc:1196] T 5019de9e275c42a697d17fc31bceac36 P 9dec645b473f40578e726f3e50e219bf: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal/wals/5019de9e275c42a697d17fc31bceac36
I20240502 21:26:49.362205 3332 ts_tablet_manager.cc:1909] T 5019de9e275c42a697d17fc31bceac36 P 9dec645b473f40578e726f3e50e219bf: Deleting consensus metadata
I20240502 21:26:49.362987 31379 catalog_manager.cc:4830] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): tablet 5019de9e275c42a697d17fc31bceac36 (table rebalancer_dynamic_table_new_name [id=df46a59a7eee43beba75e74228f448ac]) successfully deleted
I20240502 21:26:49.403525 3213 consensus_queue.cc:1035] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:49.408289 3213 consensus_queue.cc:1035] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:49.431663 2900 raft_consensus.cc:979] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c: : Instructing follower 9dec645b473f40578e726f3e50e219bf to start an election
I20240502 21:26:49.431793 2900 raft_consensus.cc:1067] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Signalling peer 9dec645b473f40578e726f3e50e219bf to start an election
I20240502 21:26:49.432263 1965 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f9bbc56ba2f841b9bb36a864e1a1f808"
dest_uuid: "9dec645b473f40578e726f3e50e219bf"
from {username='slave'} at 127.24.226.131:41999
I20240502 21:26:49.432482 1965 raft_consensus.cc:482] T f9bbc56ba2f841b9bb36a864e1a1f808 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:26:49.432590 1965 raft_consensus.cc:3037] T f9bbc56ba2f841b9bb36a864e1a1f808 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:49.435473 1965 raft_consensus.cc:504] T f9bbc56ba2f841b9bb36a864e1a1f808 P 9dec645b473f40578e726f3e50e219bf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:26:49.436769 31795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9bbc56ba2f841b9bb36a864e1a1f808" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:49.436977 31795 raft_consensus.cc:3032] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:26:49.437119 31795 raft_consensus.cc:729] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Running, Role: LEADER
I20240502 21:26:49.437119 1965 leader_election.cc:288] T f9bbc56ba2f841b9bb36a864e1a1f808 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 2 election: Requested vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:49.437433 31795 consensus_queue.cc:260] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:26:49.437803 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9bbc56ba2f841b9bb36a864e1a1f808" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:49.437934 31795 raft_consensus.cc:3037] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:49.438060 31925 raft_consensus.cc:3037] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:49.440308 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9bbc56ba2f841b9bb36a864e1a1f808" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:49.440557 31666 raft_consensus.cc:3037] T f9bbc56ba2f841b9bb36a864e1a1f808 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:49.442684 31925 raft_consensus.cc:2454] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 2.
I20240502 21:26:49.443064 31666 raft_consensus.cc:2454] T f9bbc56ba2f841b9bb36a864e1a1f808 P 05f4fac41c4f4959b9b316cb112d0ada [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 2.
I20240502 21:26:49.443324 31795 raft_consensus.cc:2454] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 2.
I20240502 21:26:49.443454 1902 leader_election.cc:302] T f9bbc56ba2f841b9bb36a864e1a1f808 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 2d50f33557944bb480ea32199d9ee4f1, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:26:49.443807 3327 raft_consensus.cc:2781] T f9bbc56ba2f841b9bb36a864e1a1f808 P 9dec645b473f40578e726f3e50e219bf [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:26:49.443994 3327 raft_consensus.cc:686] T f9bbc56ba2f841b9bb36a864e1a1f808 P 9dec645b473f40578e726f3e50e219bf [term 2 LEADER]: Becoming Leader. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Running, Role: LEADER
I20240502 21:26:49.444325 31390 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:26:49.444351 3327 consensus_queue.cc:237] T f9bbc56ba2f841b9bb36a864e1a1f808 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:26:49.447530 31389 catalog_manager.cc:5484] T f9bbc56ba2f841b9bb36a864e1a1f808 P 9dec645b473f40578e726f3e50e219bf reported cstate change: term changed from 1 to 2, leader changed from 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) to 9dec645b473f40578e726f3e50e219bf (127.24.226.134). New cstate: current_term: 2 leader_uuid: "9dec645b473f40578e726f3e50e219bf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:49.452337 31516 tablet_service.cc:1467] Processing CreateTablet for tablet 07de3b0a174e4b2da3230b049f4a7fc8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d1765af7e6b64cab9f2f4f8e8f5e8235]), partition=
I20240502 21:26:49.452603 2210 tablet_service.cc:1467] Processing CreateTablet for tablet 07de3b0a174e4b2da3230b049f4a7fc8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d1765af7e6b64cab9f2f4f8e8f5e8235]), partition=
I20240502 21:26:49.452824 31516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07de3b0a174e4b2da3230b049f4a7fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:49.452997 2210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07de3b0a174e4b2da3230b049f4a7fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:49.454200 31647 tablet_service.cc:1467] Processing CreateTablet for tablet 07de3b0a174e4b2da3230b049f4a7fc8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d1765af7e6b64cab9f2f4f8e8f5e8235]), partition=
I20240502 21:26:49.454509 31647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07de3b0a174e4b2da3230b049f4a7fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:49.460759 3224 tablet_bootstrap.cc:492] T 07de3b0a174e4b2da3230b049f4a7fc8 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:49.462188 3224 tablet_bootstrap.cc:654] T 07de3b0a174e4b2da3230b049f4a7fc8 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:49.464056 3222 tablet_bootstrap.cc:492] T 07de3b0a174e4b2da3230b049f4a7fc8 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:49.464305 3224 tablet_bootstrap.cc:492] T 07de3b0a174e4b2da3230b049f4a7fc8 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:49.464488 3224 ts_tablet_manager.cc:1367] T 07de3b0a174e4b2da3230b049f4a7fc8 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 21:26:49.465618 3222 tablet_bootstrap.cc:654] T 07de3b0a174e4b2da3230b049f4a7fc8 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:49.465550 3224 raft_consensus.cc:348] T 07de3b0a174e4b2da3230b049f4a7fc8 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:49.465759 3224 raft_consensus.cc:374] T 07de3b0a174e4b2da3230b049f4a7fc8 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:49.465829 3224 raft_consensus.cc:729] T 07de3b0a174e4b2da3230b049f4a7fc8 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:49.466135 3224 consensus_queue.cc:260] T 07de3b0a174e4b2da3230b049f4a7fc8 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:49.466679 3224 ts_tablet_manager.cc:1398] T 07de3b0a174e4b2da3230b049f4a7fc8 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:26:49.469259 3222 tablet_bootstrap.cc:492] T 07de3b0a174e4b2da3230b049f4a7fc8 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:49.469269 3338 tablet_bootstrap.cc:492] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:49.469416 3222 ts_tablet_manager.cc:1367] T 07de3b0a174e4b2da3230b049f4a7fc8 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:26:49.470191 3222 raft_consensus.cc:348] T 07de3b0a174e4b2da3230b049f4a7fc8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:49.470376 3222 raft_consensus.cc:374] T 07de3b0a174e4b2da3230b049f4a7fc8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:49.470464 3222 raft_consensus.cc:729] T 07de3b0a174e4b2da3230b049f4a7fc8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:49.470654 3222 consensus_queue.cc:260] T 07de3b0a174e4b2da3230b049f4a7fc8 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:49.472492 3338 tablet_bootstrap.cc:654] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:49.473811 3222 ts_tablet_manager.cc:1398] T 07de3b0a174e4b2da3230b049f4a7fc8 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:49.475507 3338 tablet_bootstrap.cc:492] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:49.475703 3338 ts_tablet_manager.cc:1367] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 21:26:49.476653 3338 raft_consensus.cc:348] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:49.476858 3338 raft_consensus.cc:374] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:49.476960 3338 raft_consensus.cc:729] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:49.477164 3338 consensus_queue.cc:260] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:49.477722 3338 ts_tablet_manager.cc:1398] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:49.491555 3213 consensus_queue.cc:1035] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:49.497799 2891 consensus_queue.cc:1035] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:49.501986 2631 raft_consensus.cc:482] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:49.502138 2631 raft_consensus.cc:504] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:49.502825 2631 leader_election.cc:288] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:49.503127 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07de3b0a174e4b2da3230b049f4a7fc8" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:49.503373 31666 raft_consensus.cc:2454] T 07de3b0a174e4b2da3230b049f4a7fc8 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:49.503325 2228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07de3b0a174e4b2da3230b049f4a7fc8" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:26:49.503510 2228 raft_consensus.cc:2454] T 07de3b0a174e4b2da3230b049f4a7fc8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:49.503834 31472 leader_election.cc:302] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:49.504611 2631 raft_consensus.cc:2781] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:49.504758 2631 raft_consensus.cc:482] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:49.504834 2631 raft_consensus.cc:3037] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:49.508250 2631 raft_consensus.cc:504] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:49.508827 2631 leader_election.cc:288] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Requested vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:49.509259 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07de3b0a174e4b2da3230b049f4a7fc8" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:49.509292 2228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07de3b0a174e4b2da3230b049f4a7fc8" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:26:49.509469 31666 raft_consensus.cc:3037] T 07de3b0a174e4b2da3230b049f4a7fc8 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:49.509500 2228 raft_consensus.cc:3037] T 07de3b0a174e4b2da3230b049f4a7fc8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:49.511545 2228 raft_consensus.cc:2454] T 07de3b0a174e4b2da3230b049f4a7fc8 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:49.511986 31666 raft_consensus.cc:2454] T 07de3b0a174e4b2da3230b049f4a7fc8 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:49.512172 31471 leader_election.cc:302] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a44f41b0b6644fb6aa29476aed8c964f, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:49.512835 3207 raft_consensus.cc:2781] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:49.513264 3207 raft_consensus.cc:686] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:26:49.513583 3207 consensus_queue.cc:237] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:49.516974 31389 catalog_manager.cc:5484] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 0 to 1, leader changed from <none> to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:49.623504 31389 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:26:49.628741 2228 raft_consensus.cc:1261] T 07de3b0a174e4b2da3230b049f4a7fc8 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:49.628924 31666 raft_consensus.cc:1261] T 07de3b0a174e4b2da3230b049f4a7fc8 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:49.629451 3207 consensus_queue.cc:1035] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:49.629698 2631 consensus_queue.cc:1035] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:49.633286 31389 catalog_manager.cc:5576] TS d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907) does not have the latest schema for tablet 07de3b0a174e4b2da3230b049f4a7fc8 (table rebalancer_dynamic_table [id=d1765af7e6b64cab9f2f4f8e8f5e8235]). Expected version 1 got 0
I20240502 21:26:49.633651 3245 tablet.cc:1722] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:49.634150 3245 tablet.cc:1620] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f: MemRowSet was empty: no flush needed.
I20240502 21:26:49.634192 3364 tablet.cc:1722] T 07de3b0a174e4b2da3230b049f4a7fc8 P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:49.634857 3364 tablet.cc:1620] T 07de3b0a174e4b2da3230b049f4a7fc8 P d1406e92aec84ac585304e57bd37a40d: MemRowSet was empty: no flush needed.
I20240502 21:26:49.640625 31389 catalog_manager.cc:5576] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645) does not have the latest schema for tablet 07de3b0a174e4b2da3230b049f4a7fc8 (table rebalancer_dynamic_table [id=d1765af7e6b64cab9f2f4f8e8f5e8235]). Expected version 1 got 0
I20240502 21:26:49.640869 31395 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=d1765af7e6b64cab9f2f4f8e8f5e8235] alter complete (version 1)
I20240502 21:26:49.642724 3244 tablet.cc:1722] T 07de3b0a174e4b2da3230b049f4a7fc8 P 05f4fac41c4f4959b9b316cb112d0ada: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:49.643131 3244 tablet.cc:1620] T 07de3b0a174e4b2da3230b049f4a7fc8 P 05f4fac41c4f4959b9b316cb112d0ada: MemRowSet was empty: no flush needed.
I20240502 21:26:49.746943 31389 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:26:49.751636 2974 consensus_queue.cc:1035] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:49.754209 3245 tablet.cc:1722] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:49.754760 3245 tablet.cc:1620] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f: MemRowSet was empty: no flush needed.
I20240502 21:26:49.754827 3244 tablet.cc:1722] T 07de3b0a174e4b2da3230b049f4a7fc8 P 05f4fac41c4f4959b9b316cb112d0ada: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:49.755021 3364 tablet.cc:1722] T 07de3b0a174e4b2da3230b049f4a7fc8 P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:49.755280 3244 tablet.cc:1620] T 07de3b0a174e4b2da3230b049f4a7fc8 P 05f4fac41c4f4959b9b316cb112d0ada: MemRowSet was empty: no flush needed.
I20240502 21:26:49.755415 3364 tablet.cc:1620] T 07de3b0a174e4b2da3230b049f4a7fc8 P d1406e92aec84ac585304e57bd37a40d: MemRowSet was empty: no flush needed.
I20240502 21:26:49.761353 3181 consensus_queue.cc:1035] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:49.768429 31395 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=d1765af7e6b64cab9f2f4f8e8f5e8235] alter complete (version 2)
I20240502 21:26:49.825546 31666 raft_consensus.cc:1261] T f9bbc56ba2f841b9bb36a864e1a1f808 P 05f4fac41c4f4959b9b316cb112d0ada [term 2 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:26:49.826031 3327 consensus_queue.cc:1035] T f9bbc56ba2f841b9bb36a864e1a1f808 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:49.826614 31390 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_extra_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:49.834373 1811 tablet_service.cc:1467] Processing CreateTablet for tablet acb3bdc518b84b5abc362cc4e12027c3 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=d8ac6ebe732c415bb64d1aecfeadc621]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:49.834852 1811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acb3bdc518b84b5abc362cc4e12027c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:49.834815 2075 tablet_service.cc:1467] Processing CreateTablet for tablet acb3bdc518b84b5abc362cc4e12027c3 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=d8ac6ebe732c415bb64d1aecfeadc621]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:49.835193 2075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acb3bdc518b84b5abc362cc4e12027c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:49.835446 1945 tablet_service.cc:1467] Processing CreateTablet for tablet 0d923caa3f034a899050fe62dfc0c3d8 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=d8ac6ebe732c415bb64d1aecfeadc621]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:49.835673 1944 tablet_service.cc:1467] Processing CreateTablet for tablet 04c07a003bfe472e9142df6082759ae1 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=d8ac6ebe732c415bb64d1aecfeadc621]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:49.835871 1945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d923caa3f034a899050fe62dfc0c3d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:49.836073 1944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04c07a003bfe472e9142df6082759ae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:49.839509 1946 tablet_service.cc:1467] Processing CreateTablet for tablet acb3bdc518b84b5abc362cc4e12027c3 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=d8ac6ebe732c415bb64d1aecfeadc621]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:49.839869 1946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acb3bdc518b84b5abc362cc4e12027c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:49.841537 2076 tablet_service.cc:1467] Processing CreateTablet for tablet 04c07a003bfe472e9142df6082759ae1 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=d8ac6ebe732c415bb64d1aecfeadc621]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:49.841557 2077 tablet_service.cc:1467] Processing CreateTablet for tablet 0d923caa3f034a899050fe62dfc0c3d8 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=d8ac6ebe732c415bb64d1aecfeadc621]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:49.841919 2076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04c07a003bfe472e9142df6082759ae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:49.842126 2077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d923caa3f034a899050fe62dfc0c3d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:49.845167 2210 tablet_service.cc:1467] Processing CreateTablet for tablet 0d923caa3f034a899050fe62dfc0c3d8 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=d8ac6ebe732c415bb64d1aecfeadc621]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:49.845539 2210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d923caa3f034a899050fe62dfc0c3d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:49.845439 1812 tablet_service.cc:1467] Processing CreateTablet for tablet 04c07a003bfe472e9142df6082759ae1 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=d8ac6ebe732c415bb64d1aecfeadc621]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:49.845992 1812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04c07a003bfe472e9142df6082759ae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:49.856745 3222 tablet_bootstrap.cc:492] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:49.857558 3373 tablet_bootstrap.cc:492] T 0d923caa3f034a899050fe62dfc0c3d8 P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:49.858232 3222 tablet_bootstrap.cc:654] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:49.859015 3373 tablet_bootstrap.cc:654] T 0d923caa3f034a899050fe62dfc0c3d8 P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:49.859411 3374 tablet_bootstrap.cc:492] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:49.860180 3372 tablet_bootstrap.cc:492] T acb3bdc518b84b5abc362cc4e12027c3 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:49.861661 3374 tablet_bootstrap.cc:654] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:49.862008 3372 tablet_bootstrap.cc:654] T acb3bdc518b84b5abc362cc4e12027c3 P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:49.864392 3373 tablet_bootstrap.cc:492] T 0d923caa3f034a899050fe62dfc0c3d8 P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:26:49.864567 3373 ts_tablet_manager.cc:1367] T 0d923caa3f034a899050fe62dfc0c3d8 P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240502 21:26:49.865211 3373 raft_consensus.cc:348] T 0d923caa3f034a899050fe62dfc0c3d8 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:49.865374 3373 raft_consensus.cc:374] T 0d923caa3f034a899050fe62dfc0c3d8 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:49.865443 3373 raft_consensus.cc:729] T 0d923caa3f034a899050fe62dfc0c3d8 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:26:49.865610 3373 consensus_queue.cc:260] T 0d923caa3f034a899050fe62dfc0c3d8 P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:49.866042 3373 ts_tablet_manager.cc:1398] T 0d923caa3f034a899050fe62dfc0c3d8 P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:26:49.866221 3373 tablet_bootstrap.cc:492] T 04c07a003bfe472e9142df6082759ae1 P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:49.867343 3373 tablet_bootstrap.cc:654] T 04c07a003bfe472e9142df6082759ae1 P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:49.868995 3373 tablet_bootstrap.cc:492] T 04c07a003bfe472e9142df6082759ae1 P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:26:49.869261 3373 ts_tablet_manager.cc:1367] T 04c07a003bfe472e9142df6082759ae1 P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:49.870005 3373 raft_consensus.cc:348] T 04c07a003bfe472e9142df6082759ae1 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:49.870297 3373 raft_consensus.cc:374] T 04c07a003bfe472e9142df6082759ae1 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:49.870435 3373 raft_consensus.cc:729] T 04c07a003bfe472e9142df6082759ae1 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:26:49.870710 3373 consensus_queue.cc:260] T 04c07a003bfe472e9142df6082759ae1 P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:49.871269 3373 ts_tablet_manager.cc:1398] T 04c07a003bfe472e9142df6082759ae1 P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:49.871573 3373 tablet_bootstrap.cc:492] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:49.872838 3222 tablet_bootstrap.cc:492] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:49.873075 3373 tablet_bootstrap.cc:654] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:49.873168 3222 ts_tablet_manager.cc:1367] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.003s
I20240502 21:26:49.874290 3374 tablet_bootstrap.cc:492] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:49.874190 3222 raft_consensus.cc:348] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:49.874469 3374 ts_tablet_manager.cc:1367] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.004s
I20240502 21:26:49.874652 3222 raft_consensus.cc:374] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:49.874825 3222 raft_consensus.cc:729] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:49.875306 3374 raft_consensus.cc:348] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:49.875414 31395 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:26:49.875583 3374 raft_consensus.cc:374] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:49.875728 3374 raft_consensus.cc:729] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:49.875977 3374 consensus_queue.cc:260] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:49.876616 3374 ts_tablet_manager.cc:1398] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:49.876892 3374 tablet_bootstrap.cc:492] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:49.876785 3222 consensus_queue.cc:260] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:49.879233 3374 tablet_bootstrap.cc:654] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:49.882905 3373 tablet_bootstrap.cc:492] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:26:49.883041 3373 ts_tablet_manager.cc:1367] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.004s
I20240502 21:26:49.879202 3222 ts_tablet_manager.cc:1398] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:26:49.884402 3373 raft_consensus.cc:348] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:49.884712 3373 raft_consensus.cc:374] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:49.884824 3373 raft_consensus.cc:729] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:26:49.885061 3373 consensus_queue.cc:260] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:49.885417 3245 tablet.cc:1722] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:49.886984 3373 ts_tablet_manager.cc:1398] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 21:26:49.890236 3364 tablet.cc:1722] T 07de3b0a174e4b2da3230b049f4a7fc8 P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:49.897689 3374 tablet_bootstrap.cc:492] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:49.897876 3374 ts_tablet_manager.cc:1367] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.007s
I20240502 21:26:49.898054 3244 tablet.cc:1722] T 07de3b0a174e4b2da3230b049f4a7fc8 P 05f4fac41c4f4959b9b316cb112d0ada: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:49.898731 3374 raft_consensus.cc:348] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:49.899089 3374 raft_consensus.cc:374] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:49.899269 3374 raft_consensus.cc:729] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:49.899590 3374 consensus_queue.cc:260] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:49.900107 3372 tablet_bootstrap.cc:492] T acb3bdc518b84b5abc362cc4e12027c3 P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:49.900290 3372 ts_tablet_manager.cc:1367] T acb3bdc518b84b5abc362cc4e12027c3 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.040s user 0.008s sys 0.008s
I20240502 21:26:49.901271 3374 ts_tablet_manager.cc:1398] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:26:49.901948 3372 raft_consensus.cc:348] T acb3bdc518b84b5abc362cc4e12027c3 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:49.902159 3372 raft_consensus.cc:374] T acb3bdc518b84b5abc362cc4e12027c3 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:49.902272 3372 raft_consensus.cc:729] T acb3bdc518b84b5abc362cc4e12027c3 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:49.902498 3372 consensus_queue.cc:260] T acb3bdc518b84b5abc362cc4e12027c3 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:49.903038 3372 ts_tablet_manager.cc:1398] T acb3bdc518b84b5abc362cc4e12027c3 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:26:49.903339 3372 tablet_bootstrap.cc:492] T 0d923caa3f034a899050fe62dfc0c3d8 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:49.904661 31389 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table_new_name [id=d1765af7e6b64cab9f2f4f8e8f5e8235] alter complete (version 3)
I20240502 21:26:49.905185 3372 tablet_bootstrap.cc:654] T 0d923caa3f034a899050fe62dfc0c3d8 P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:49.910840 3372 tablet_bootstrap.cc:492] T 0d923caa3f034a899050fe62dfc0c3d8 P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:49.910989 3372 ts_tablet_manager.cc:1367] T 0d923caa3f034a899050fe62dfc0c3d8 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20240502 21:26:49.911532 3372 raft_consensus.cc:348] T 0d923caa3f034a899050fe62dfc0c3d8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:49.911700 3372 raft_consensus.cc:374] T 0d923caa3f034a899050fe62dfc0c3d8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:49.911777 3372 raft_consensus.cc:729] T 0d923caa3f034a899050fe62dfc0c3d8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:49.911981 3372 consensus_queue.cc:260] T 0d923caa3f034a899050fe62dfc0c3d8 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:49.912411 3372 ts_tablet_manager.cc:1398] T 0d923caa3f034a899050fe62dfc0c3d8 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:26:49.912683 3372 tablet_bootstrap.cc:492] T 04c07a003bfe472e9142df6082759ae1 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:49.914083 3372 tablet_bootstrap.cc:654] T 04c07a003bfe472e9142df6082759ae1 P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:49.916889 3372 tablet_bootstrap.cc:492] T 04c07a003bfe472e9142df6082759ae1 P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:49.917004 3372 ts_tablet_manager.cc:1367] T 04c07a003bfe472e9142df6082759ae1 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240502 21:26:49.917620 3372 raft_consensus.cc:348] T 04c07a003bfe472e9142df6082759ae1 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:49.917860 3372 raft_consensus.cc:374] T 04c07a003bfe472e9142df6082759ae1 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:49.917966 3372 raft_consensus.cc:729] T 04c07a003bfe472e9142df6082759ae1 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:49.918154 3372 consensus_queue.cc:260] T 04c07a003bfe472e9142df6082759ae1 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:49.918612 3372 ts_tablet_manager.cc:1398] T 04c07a003bfe472e9142df6082759ae1 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:49.940523 3327 raft_consensus.cc:482] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:49.940685 3327 raft_consensus.cc:504] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:49.941231 3327 leader_election.cc:288] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:49.941287 2429 raft_consensus.cc:482] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:49.941486 2429 raft_consensus.cc:504] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:49.941989 2429 leader_election.cc:288] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:49.942862 1966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d923caa3f034a899050fe62dfc0c3d8" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:26:49.943071 1966 raft_consensus.cc:2454] T 0d923caa3f034a899050fe62dfc0c3d8 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:26:49.943117 1833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3bdc518b84b5abc362cc4e12027c3" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:26:49.943359 1833 raft_consensus.cc:2454] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 0.
I20240502 21:26:49.943343 2097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3bdc518b84b5abc362cc4e12027c3" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:49.943583 2097 raft_consensus.cc:2454] T acb3bdc518b84b5abc362cc4e12027c3 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 0.
I20240502 21:26:49.943753 2816 raft_consensus.cc:482] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:49.943905 2816 raft_consensus.cc:504] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:49.944294 1901 leader_election.cc:302] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:26:49.944649 2816 leader_election.cc:288] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:26:49.944643 2161 leader_election.cc:302] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dec645b473f40578e726f3e50e219bf, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:49.945044 2429 raft_consensus.cc:2781] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:49.945149 2429 raft_consensus.cc:482] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:49.945255 2429 raft_consensus.cc:3037] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:49.946034 2095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d923caa3f034a899050fe62dfc0c3d8" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:49.946259 2095 raft_consensus.cc:2454] T 0d923caa3f034a899050fe62dfc0c3d8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:26:49.947549 2429 raft_consensus.cc:504] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:49.949453 2097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04c07a003bfe472e9142df6082759ae1" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:49.949649 2097 raft_consensus.cc:2454] T 04c07a003bfe472e9142df6082759ae1 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 0.
I20240502 21:26:49.950058 1769 leader_election.cc:302] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 1e04157b98dd4fad98858c290b73c3db; no voters:
I20240502 21:26:49.950660 2816 raft_consensus.cc:2781] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:49.950773 2816 raft_consensus.cc:482] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:49.950847 2816 raft_consensus.cc:3037] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:49.951828 2097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d923caa3f034a899050fe62dfc0c3d8" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:26:49.951995 2097 raft_consensus.cc:3037] T 0d923caa3f034a899050fe62dfc0c3d8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:49.952651 3327 raft_consensus.cc:2781] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:49.952739 3327 raft_consensus.cc:482] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:49.952826 3327 raft_consensus.cc:3037] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:49.953276 1965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04c07a003bfe472e9142df6082759ae1" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:26:49.953282 2816 raft_consensus.cc:504] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:49.953477 1965 raft_consensus.cc:2454] T 04c07a003bfe472e9142df6082759ae1 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 0.
I20240502 21:26:49.953856 1966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d923caa3f034a899050fe62dfc0c3d8" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf"
I20240502 21:26:49.953938 2816 leader_election.cc:288] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Requested vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:26:49.954075 1966 raft_consensus.cc:3037] T 0d923caa3f034a899050fe62dfc0c3d8 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:49.955538 3327 raft_consensus.cc:504] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:49.956141 3327 leader_election.cc:288] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 election: Requested vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:49.956746 31925 raft_consensus.cc:1261] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 [term 2 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:26:49.958388 2095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04c07a003bfe472e9142df6082759ae1" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:26:49.958590 2095 raft_consensus.cc:3037] T 04c07a003bfe472e9142df6082759ae1 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:49.958559 1965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04c07a003bfe472e9142df6082759ae1" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf"
I20240502 21:26:49.958729 1965 raft_consensus.cc:3037] T 04c07a003bfe472e9142df6082759ae1 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:49.961160 2095 raft_consensus.cc:2454] T 04c07a003bfe472e9142df6082759ae1 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:26:49.958395 2096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3bdc518b84b5abc362cc4e12027c3" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:26:49.961576 2096 raft_consensus.cc:3037] T acb3bdc518b84b5abc362cc4e12027c3 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:49.962889 2097 raft_consensus.cc:2454] T 0d923caa3f034a899050fe62dfc0c3d8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 1.
I20240502 21:26:49.963480 2163 leader_election.cc:302] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:49.963788 2891 raft_consensus.cc:2781] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:49.963922 2891 raft_consensus.cc:686] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Becoming Leader. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Running, Role: LEADER
I20240502 21:26:49.964185 2096 raft_consensus.cc:2454] T acb3bdc518b84b5abc362cc4e12027c3 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 1.
I20240502 21:26:49.964201 2891 consensus_queue.cc:237] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:49.964510 1965 raft_consensus.cc:2454] T 04c07a003bfe472e9142df6082759ae1 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:26:49.965031 1767 leader_election.cc:302] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:26:49.965591 3066 raft_consensus.cc:2781] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:49.965727 3066 raft_consensus.cc:686] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Becoming Leader. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Running, Role: LEADER
I20240502 21:26:49.965974 3066 consensus_queue.cc:237] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:49.966269 3327 consensus_queue.cc:1035] T f9bbc56ba2f841b9bb36a864e1a1f808 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:49.970752 1833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3bdc518b84b5abc362cc4e12027c3" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:26:49.970943 1833 raft_consensus.cc:3037] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:49.973245 1966 raft_consensus.cc:2454] T 0d923caa3f034a899050fe62dfc0c3d8 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 1.
I20240502 21:26:49.973568 2429 leader_election.cc:288] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Requested vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:49.976636 1833 raft_consensus.cc:2454] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 1.
I20240502 21:26:49.977362 1901 leader_election.cc:302] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:26:49.978503 31390 catalog_manager.cc:5484] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d reported cstate change: term changed from 0 to 1, leader changed from <none> to d1406e92aec84ac585304e57bd37a40d (127.24.226.136). New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:49.979415 31395 catalog_manager.cc:5484] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e04157b98dd4fad98858c290b73c3db (127.24.226.133). New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:49.989918 3327 raft_consensus.cc:2781] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:49.990041 3327 raft_consensus.cc:686] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf [term 1 LEADER]: Becoming Leader. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Running, Role: LEADER
I20240502 21:26:49.990314 3327 consensus_queue.cc:237] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:49.997627 31795 raft_consensus.cc:1261] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [term 2 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:26:49.998217 3327 consensus_queue.cc:1035] T f9bbc56ba2f841b9bb36a864e1a1f808 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:49.999815 31395 catalog_manager.cc:5484] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dec645b473f40578e726f3e50e219bf (127.24.226.134). New cstate: current_term: 1 leader_uuid: "9dec645b473f40578e726f3e50e219bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:50.008952 1966 consensus_queue.cc:237] T f9bbc56ba2f841b9bb36a864e1a1f808 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:26:50.009037 31393 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:50.009250 31393 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:50.011998 31666 raft_consensus.cc:1261] T f9bbc56ba2f841b9bb36a864e1a1f808 P 05f4fac41c4f4959b9b316cb112d0ada [term 2 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 21:26:50.012147 31925 raft_consensus.cc:1261] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 [term 2 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 21:26:50.014652 31393 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 07de3b0a174e4b2da3230b049f4a7fc8
I20240502 21:26:50.014144 3398 consensus_queue.cc:1035] T f9bbc56ba2f841b9bb36a864e1a1f808 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240502 21:26:50.015400 31647 tablet_service.cc:1514] Processing DeleteTablet for tablet 07de3b0a174e4b2da3230b049f4a7fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:50 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:50.015600 3327 consensus_queue.cc:1035] T f9bbc56ba2f841b9bb36a864e1a1f808 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:26:50.015662 2210 tablet_service.cc:1514] Processing DeleteTablet for tablet 07de3b0a174e4b2da3230b049f4a7fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:50 UTC) from {username='slave'} at 127.0.0.1:36984
I20240502 21:26:50.016280 31516 tablet_service.cc:1514] Processing DeleteTablet for tablet 07de3b0a174e4b2da3230b049f4a7fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:50 UTC) from {username='slave'} at 127.0.0.1:45194
I20240502 21:26:50.021277 3392 raft_consensus.cc:2932] T f9bbc56ba2f841b9bb36a864e1a1f808 P 9dec645b473f40578e726f3e50e219bf [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:50.021786 31925 raft_consensus.cc:2932] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:50.028895 3401 tablet_replica.cc:330] T 07de3b0a174e4b2da3230b049f4a7fc8 P d1406e92aec84ac585304e57bd37a40d: stopping tablet replica
I20240502 21:26:50.028512 31393 catalog_manager.cc:5484] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: config changed from index 5 to 7, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New cstate: current_term: 2 leader_uuid: "9dec645b473f40578e726f3e50e219bf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:50.029136 3401 raft_consensus.cc:2229] T 07de3b0a174e4b2da3230b049f4a7fc8 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:50.029354 3401 raft_consensus.cc:2258] T 07de3b0a174e4b2da3230b049f4a7fc8 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:50.030308 3401 ts_tablet_manager.cc:1875] T 07de3b0a174e4b2da3230b049f4a7fc8 P d1406e92aec84ac585304e57bd37a40d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:50.035434 3401 ts_tablet_manager.cc:1888] T 07de3b0a174e4b2da3230b049f4a7fc8 P d1406e92aec84ac585304e57bd37a40d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:50.035555 3401 log.cc:1196] T 07de3b0a174e4b2da3230b049f4a7fc8 P d1406e92aec84ac585304e57bd37a40d: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal/wals/07de3b0a174e4b2da3230b049f4a7fc8
I20240502 21:26:50.035945 3401 ts_tablet_manager.cc:1909] T 07de3b0a174e4b2da3230b049f4a7fc8 P d1406e92aec84ac585304e57bd37a40d: Deleting consensus metadata
I20240502 21:26:50.036948 31775 tablet_service.cc:1514] Processing DeleteTablet for tablet f9bbc56ba2f841b9bb36a864e1a1f808 with delete_type TABLET_DATA_TOMBSTONED (TS 08d7a0680d244b55b93d17f9ab57b84c not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51596
I20240502 21:26:50.037233 31381 catalog_manager.cc:4830] TS d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): tablet 07de3b0a174e4b2da3230b049f4a7fc8 (table rebalancer_dynamic_table_new_name [id=d1765af7e6b64cab9f2f4f8e8f5e8235]) successfully deleted
I20240502 21:26:50.043694 31379 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f9bbc56ba2f841b9bb36a864e1a1f808 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:50.045532 3402 tablet_replica.cc:330] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:50.045833 3402 raft_consensus.cc:2229] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:50.046241 3402 raft_consensus.cc:2258] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:50.046901 3400 tablet_replica.cc:330] T 07de3b0a174e4b2da3230b049f4a7fc8 P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:50.046630 31666 raft_consensus.cc:2932] T f9bbc56ba2f841b9bb36a864e1a1f808 P 05f4fac41c4f4959b9b316cb112d0ada [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:50.047461 3405 tablet_replica.cc:330] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c: stopping tablet replica
I20240502 21:26:50.047647 3405 raft_consensus.cc:2229] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:50.047837 3405 raft_consensus.cc:2258] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:50.048672 3402 ts_tablet_manager.cc:1875] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:50.048672 3400 raft_consensus.cc:2229] T 07de3b0a174e4b2da3230b049f4a7fc8 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:50.048914 3400 raft_consensus.cc:2258] T 07de3b0a174e4b2da3230b049f4a7fc8 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:50.050778 3400 ts_tablet_manager.cc:1875] T 07de3b0a174e4b2da3230b049f4a7fc8 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:50.053161 3405 ts_tablet_manager.cc:1875] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:50.055846 3400 ts_tablet_manager.cc:1888] T 07de3b0a174e4b2da3230b049f4a7fc8 P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:50.055938 3400 log.cc:1196] T 07de3b0a174e4b2da3230b049f4a7fc8 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/07de3b0a174e4b2da3230b049f4a7fc8
I20240502 21:26:50.056298 3400 ts_tablet_manager.cc:1909] T 07de3b0a174e4b2da3230b049f4a7fc8 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:26:50.057225 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet 07de3b0a174e4b2da3230b049f4a7fc8 (table rebalancer_dynamic_table_new_name [id=d1765af7e6b64cab9f2f4f8e8f5e8235]) successfully deleted
I20240502 21:26:50.058743 3405 ts_tablet_manager.cc:1888] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 21:26:50.058921 3405 log.cc:1196] T f9bbc56ba2f841b9bb36a864e1a1f808 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/wals/f9bbc56ba2f841b9bb36a864e1a1f808
I20240502 21:26:50.059682 31381 catalog_manager.cc:4830] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): tablet f9bbc56ba2f841b9bb36a864e1a1f808 (table rebalance_test_table_7 [id=0af22f3605164ff58dc3914737c2de4f]) successfully deleted
I20240502 21:26:50.060312 3402 ts_tablet_manager.cc:1888] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:50.060423 3402 log.cc:1196] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/07de3b0a174e4b2da3230b049f4a7fc8
I20240502 21:26:50.060838 3402 ts_tablet_manager.cc:1909] T 07de3b0a174e4b2da3230b049f4a7fc8 P a44f41b0b6644fb6aa29476aed8c964f: Deleting consensus metadata
I20240502 21:26:50.061722 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet 07de3b0a174e4b2da3230b049f4a7fc8 (table rebalancer_dynamic_table_new_name [id=d1765af7e6b64cab9f2f4f8e8f5e8235]) successfully deleted
I20240502 21:26:50.118991 31395 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:26:50.126341 2075 tablet_service.cc:1467] Processing CreateTablet for tablet aca362d1a6894071b195dd9fb7594b1f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2faf275486e74ad19118c3628524b2e6]), partition=
I20240502 21:26:50.126716 2075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aca362d1a6894071b195dd9fb7594b1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:50.126801 2210 tablet_service.cc:1467] Processing CreateTablet for tablet aca362d1a6894071b195dd9fb7594b1f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2faf275486e74ad19118c3628524b2e6]), partition=
I20240502 21:26:50.127110 2210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aca362d1a6894071b195dd9fb7594b1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:50.128773 31647 tablet_service.cc:1467] Processing CreateTablet for tablet aca362d1a6894071b195dd9fb7594b1f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2faf275486e74ad19118c3628524b2e6]), partition=
I20240502 21:26:50.129132 31647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aca362d1a6894071b195dd9fb7594b1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:50.132229 3372 tablet_bootstrap.cc:492] T aca362d1a6894071b195dd9fb7594b1f P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:50.133577 3372 tablet_bootstrap.cc:654] T aca362d1a6894071b195dd9fb7594b1f P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:50.135793 3408 tablet_bootstrap.cc:492] T aca362d1a6894071b195dd9fb7594b1f P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:50.137753 3408 tablet_bootstrap.cc:654] T aca362d1a6894071b195dd9fb7594b1f P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:50.138303 3222 tablet_bootstrap.cc:492] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:50.140112 3222 tablet_bootstrap.cc:654] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:50.143474 3372 tablet_bootstrap.cc:492] T aca362d1a6894071b195dd9fb7594b1f P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:50.143728 3372 ts_tablet_manager.cc:1367] T aca362d1a6894071b195dd9fb7594b1f P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20240502 21:26:50.144699 3372 raft_consensus.cc:348] T aca362d1a6894071b195dd9fb7594b1f P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:50.144987 3372 raft_consensus.cc:374] T aca362d1a6894071b195dd9fb7594b1f P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:50.145067 3372 raft_consensus.cc:729] T aca362d1a6894071b195dd9fb7594b1f P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:50.145397 3372 consensus_queue.cc:260] T aca362d1a6894071b195dd9fb7594b1f P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:50.146016 3372 ts_tablet_manager.cc:1398] T aca362d1a6894071b195dd9fb7594b1f P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:50.148525 3222 tablet_bootstrap.cc:492] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:50.148756 3222 ts_tablet_manager.cc:1367] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20240502 21:26:50.149489 3222 raft_consensus.cc:348] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:50.149698 3222 raft_consensus.cc:374] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:50.149825 3222 raft_consensus.cc:729] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:50.150056 3222 consensus_queue.cc:260] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:50.151234 3222 ts_tablet_manager.cc:1398] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:50.169773 3408 tablet_bootstrap.cc:492] T aca362d1a6894071b195dd9fb7594b1f P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:50.170048 3408 ts_tablet_manager.cc:1367] T aca362d1a6894071b195dd9fb7594b1f P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.024s
I20240502 21:26:50.170845 3408 raft_consensus.cc:348] T aca362d1a6894071b195dd9fb7594b1f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:50.171070 3408 raft_consensus.cc:374] T aca362d1a6894071b195dd9fb7594b1f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:50.171165 3408 raft_consensus.cc:729] T aca362d1a6894071b195dd9fb7594b1f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:50.171352 3408 consensus_queue.cc:260] T aca362d1a6894071b195dd9fb7594b1f P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:50.171967 3408 ts_tablet_manager.cc:1398] T aca362d1a6894071b195dd9fb7594b1f P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:26:50.186304 2429 raft_consensus.cc:482] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:50.186565 2429 raft_consensus.cc:504] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:50.187371 2429 leader_election.cc:288] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:50.188074 2095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca362d1a6894071b195dd9fb7594b1f" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:50.188303 2095 raft_consensus.cc:2454] T aca362d1a6894071b195dd9fb7594b1f P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:26:50.188738 2163 leader_election.cc:302] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:50.190779 2429 raft_consensus.cc:2781] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:50.190862 2429 raft_consensus.cc:482] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:50.190996 2429 raft_consensus.cc:3037] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:50.192720 2429 raft_consensus.cc:504] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:50.193384 2429 leader_election.cc:288] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Requested vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:50.193796 2095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca362d1a6894071b195dd9fb7594b1f" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:26:50.193964 2095 raft_consensus.cc:3037] T aca362d1a6894071b195dd9fb7594b1f P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:50.195865 2095 raft_consensus.cc:2454] T aca362d1a6894071b195dd9fb7594b1f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 1.
I20240502 21:26:50.196205 2163 leader_election.cc:302] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:50.196465 2429 raft_consensus.cc:2781] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:50.196667 2429 raft_consensus.cc:686] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Becoming Leader. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Running, Role: LEADER
I20240502 21:26:50.197016 2429 consensus_queue.cc:237] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:50.200351 31390 catalog_manager.cc:5484] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d reported cstate change: term changed from 0 to 1, leader changed from <none> to d1406e92aec84ac585304e57bd37a40d (127.24.226.136). New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:50.208091 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca362d1a6894071b195dd9fb7594b1f" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:50.208284 31666 raft_consensus.cc:2454] T aca362d1a6894071b195dd9fb7594b1f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:26:50.208252 31667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca362d1a6894071b195dd9fb7594b1f" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:50.208510 31667 raft_consensus.cc:3037] T aca362d1a6894071b195dd9fb7594b1f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:50.211994 31667 raft_consensus.cc:2454] T aca362d1a6894071b195dd9fb7594b1f P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 1.
I20240502 21:26:50.310678 31394 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:26:50.318125 31667 raft_consensus.cc:1261] T aca362d1a6894071b195dd9fb7594b1f P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:50.318162 2095 raft_consensus.cc:1261] T aca362d1a6894071b195dd9fb7594b1f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:50.318733 3213 consensus_queue.cc:1035] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:50.319193 3213 consensus_queue.cc:1035] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:50.324743 31391 catalog_manager.cc:5576] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569) does not have the latest schema for tablet aca362d1a6894071b195dd9fb7594b1f (table rebalancer_dynamic_table [id=2faf275486e74ad19118c3628524b2e6]). Expected version 1 got 0
I20240502 21:26:50.324709 3364 tablet.cc:1722] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:50.325201 3364 tablet.cc:1620] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d: MemRowSet was empty: no flush needed.
I20240502 21:26:50.334918 3451 tablet.cc:1722] T aca362d1a6894071b195dd9fb7594b1f P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:50.335448 3451 tablet.cc:1620] T aca362d1a6894071b195dd9fb7594b1f P 1ade0354c44c4c37ae480c7a94b11e1b: MemRowSet was empty: no flush needed.
I20240502 21:26:50.342146 31389 catalog_manager.cc:5576] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569) does not have the latest schema for tablet aca362d1a6894071b195dd9fb7594b1f (table rebalancer_dynamic_table [id=2faf275486e74ad19118c3628524b2e6]). Expected version 1 got 0
I20240502 21:26:50.345319 31395 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=2faf275486e74ad19118c3628524b2e6] alter complete (version 1)
I20240502 21:26:50.351576 3066 consensus_queue.cc:1035] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:50.359951 31395 catalog_manager.cc:5576] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645) does not have the latest schema for tablet aca362d1a6894071b195dd9fb7594b1f (table rebalancer_dynamic_table [id=2faf275486e74ad19118c3628524b2e6]). Expected version 1 got 0
I20240502 21:26:50.360620 3244 tablet.cc:1722] T aca362d1a6894071b195dd9fb7594b1f P 05f4fac41c4f4959b9b316cb112d0ada: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:50.361282 3244 tablet.cc:1620] T aca362d1a6894071b195dd9fb7594b1f P 05f4fac41c4f4959b9b316cb112d0ada: MemRowSet was empty: no flush needed.
I20240502 21:26:50.392563 2816 consensus_queue.cc:1035] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:50.394039 3404 consensus_queue.cc:1035] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:50.403184 3394 consensus_queue.cc:1035] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:50.444801 31775 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 21:26:50.445714 31647 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:26:50.447479 31395 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:26:50.456676 3364 tablet.cc:1722] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:50.457211 3364 tablet.cc:1620] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d: MemRowSet was empty: no flush needed.
I20240502 21:26:50.461138 3244 tablet.cc:1722] T aca362d1a6894071b195dd9fb7594b1f P 05f4fac41c4f4959b9b316cb112d0ada: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:50.461609 3244 tablet.cc:1620] T aca362d1a6894071b195dd9fb7594b1f P 05f4fac41c4f4959b9b316cb112d0ada: MemRowSet was empty: no flush needed.
I20240502 21:26:50.461553 3451 tablet.cc:1722] T aca362d1a6894071b195dd9fb7594b1f P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:50.462007 3451 tablet.cc:1620] T aca362d1a6894071b195dd9fb7594b1f P 1ade0354c44c4c37ae480c7a94b11e1b: MemRowSet was empty: no flush needed.
I20240502 21:26:50.471165 31395 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=2faf275486e74ad19118c3628524b2e6] alter complete (version 2)
I20240502 21:26:50.476624 1811 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:26:50.490412 3213 consensus_queue.cc:1035] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:50.499871 1945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:26:50.531694 2429 consensus_queue.cc:1035] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:50.536026 31516 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 21:26:50.537350 31395 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_extra_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:50.541185 31905 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:26:50.546190 2075 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:50.562122 2075 tablet_service.cc:1467] Processing CreateTablet for tablet 4117f23449114eaa814df89f80223e5e (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=cc412415870b4447b2e9707c2f7fd323]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:50.562543 2075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4117f23449114eaa814df89f80223e5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:50.562659 1811 tablet_service.cc:1467] Processing CreateTablet for tablet 4117f23449114eaa814df89f80223e5e (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=cc412415870b4447b2e9707c2f7fd323]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:50.563010 1811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4117f23449114eaa814df89f80223e5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:50.563288 2210 tablet_service.cc:1467] Processing CreateTablet for tablet 883c729aa26f47178f40a17ad3b7e590 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=cc412415870b4447b2e9707c2f7fd323]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:50.563678 2210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 883c729aa26f47178f40a17ad3b7e590. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:50.564780 1812 tablet_service.cc:1467] Processing CreateTablet for tablet 883c729aa26f47178f40a17ad3b7e590 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=cc412415870b4447b2e9707c2f7fd323]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:50.565084 2076 tablet_service.cc:1467] Processing CreateTablet for tablet 883c729aa26f47178f40a17ad3b7e590 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=cc412415870b4447b2e9707c2f7fd323]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:50.565162 1812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 883c729aa26f47178f40a17ad3b7e590. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:50.565393 2076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 883c729aa26f47178f40a17ad3b7e590. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:50.571054 3372 tablet_bootstrap.cc:492] T 883c729aa26f47178f40a17ad3b7e590 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:50.572630 3372 tablet_bootstrap.cc:654] T 883c729aa26f47178f40a17ad3b7e590 P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:50.576138 31647 tablet_service.cc:1467] Processing CreateTablet for tablet 4117f23449114eaa814df89f80223e5e (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=cc412415870b4447b2e9707c2f7fd323]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:50.576617 3222 tablet_bootstrap.cc:492] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:50.578344 3222 tablet_bootstrap.cc:654] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:50.580099 31647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4117f23449114eaa814df89f80223e5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:50.581507 31389 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:26:50.583789 2210 tablet_service.cc:1467] Processing CreateTablet for tablet 51551a1967b74094b373cc030e2bf850 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=cc412415870b4447b2e9707c2f7fd323]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:50.583969 1945 tablet_service.cc:1467] Processing CreateTablet for tablet 51551a1967b74094b373cc030e2bf850 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=cc412415870b4447b2e9707c2f7fd323]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:50.584225 2210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51551a1967b74094b373cc030e2bf850. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:50.584410 1945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51551a1967b74094b373cc030e2bf850. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:50.587256 31645 tablet_service.cc:1467] Processing CreateTablet for tablet 51551a1967b74094b373cc030e2bf850 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=cc412415870b4447b2e9707c2f7fd323]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:50.587637 31645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51551a1967b74094b373cc030e2bf850. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:50.593300 3408 tablet_bootstrap.cc:492] T 51551a1967b74094b373cc030e2bf850 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:50.593686 3364 tablet.cc:1722] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:50.595351 3408 tablet_bootstrap.cc:654] T 51551a1967b74094b373cc030e2bf850 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:50.599071 3451 tablet.cc:1722] T aca362d1a6894071b195dd9fb7594b1f P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:50.610320 3244 tablet.cc:1722] T aca362d1a6894071b195dd9fb7594b1f P 05f4fac41c4f4959b9b316cb112d0ada: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:50.618582 3408 tablet_bootstrap.cc:492] T 51551a1967b74094b373cc030e2bf850 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:50.618830 3408 ts_tablet_manager.cc:1367] T 51551a1967b74094b373cc030e2bf850 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.008s
I20240502 21:26:50.619346 2208 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:26:50.620028 3408 raft_consensus.cc:348] T 51551a1967b74094b373cc030e2bf850 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:50.620369 3408 raft_consensus.cc:374] T 51551a1967b74094b373cc030e2bf850 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:50.620736 3408 raft_consensus.cc:729] T 51551a1967b74094b373cc030e2bf850 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:50.621152 3408 consensus_queue.cc:260] T 51551a1967b74094b373cc030e2bf850 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:50.621856 3408 ts_tablet_manager.cc:1398] T 51551a1967b74094b373cc030e2bf850 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 21:26:50.622678 3408 tablet_bootstrap.cc:492] T 4117f23449114eaa814df89f80223e5e P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:50.624887 3408 tablet_bootstrap.cc:654] T 4117f23449114eaa814df89f80223e5e P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:50.636552 31389 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table_new_name [id=2faf275486e74ad19118c3628524b2e6] alter complete (version 3)
I20240502 21:26:50.648490 3222 tablet_bootstrap.cc:492] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:50.649412 3222 ts_tablet_manager.cc:1367] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.072s user 0.006s sys 0.004s
I20240502 21:26:50.650477 3222 raft_consensus.cc:348] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:50.652455 3222 raft_consensus.cc:374] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:50.652580 3222 raft_consensus.cc:729] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:50.652905 3222 consensus_queue.cc:260] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:50.654868 3222 ts_tablet_manager.cc:1398] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 21:26:50.655344 3222 tablet_bootstrap.cc:492] T 51551a1967b74094b373cc030e2bf850 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:50.658793 3408 tablet_bootstrap.cc:492] T 4117f23449114eaa814df89f80223e5e P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:50.658965 3408 ts_tablet_manager.cc:1367] T 4117f23449114eaa814df89f80223e5e P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.001s
I20240502 21:26:50.659636 3408 raft_consensus.cc:348] T 4117f23449114eaa814df89f80223e5e P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:50.659900 3408 raft_consensus.cc:374] T 4117f23449114eaa814df89f80223e5e P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:50.660044 3408 raft_consensus.cc:729] T 4117f23449114eaa814df89f80223e5e P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:50.660298 3408 consensus_queue.cc:260] T 4117f23449114eaa814df89f80223e5e P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:50.660820 3222 tablet_bootstrap.cc:654] T 51551a1967b74094b373cc030e2bf850 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:50.661012 3408 ts_tablet_manager.cc:1398] T 4117f23449114eaa814df89f80223e5e P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:50.669821 3479 tablet_bootstrap.cc:492] T 883c729aa26f47178f40a17ad3b7e590 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:50.671425 3479 tablet_bootstrap.cc:654] T 883c729aa26f47178f40a17ad3b7e590 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:50.679790 3481 tablet_bootstrap.cc:492] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:50.680976 3372 tablet_bootstrap.cc:492] T 883c729aa26f47178f40a17ad3b7e590 P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:50.681149 3372 ts_tablet_manager.cc:1367] T 883c729aa26f47178f40a17ad3b7e590 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.110s user 0.006s sys 0.012s
I20240502 21:26:50.681854 3372 raft_consensus.cc:348] T 883c729aa26f47178f40a17ad3b7e590 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:50.682051 3372 raft_consensus.cc:374] T 883c729aa26f47178f40a17ad3b7e590 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:50.682145 3372 raft_consensus.cc:729] T 883c729aa26f47178f40a17ad3b7e590 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:50.686452 3372 consensus_queue.cc:260] T 883c729aa26f47178f40a17ad3b7e590 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:50.687086 3372 ts_tablet_manager.cc:1398] T 883c729aa26f47178f40a17ad3b7e590 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 21:26:50.687340 3372 tablet_bootstrap.cc:492] T 4117f23449114eaa814df89f80223e5e P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:50.689906 3372 tablet_bootstrap.cc:654] T 4117f23449114eaa814df89f80223e5e P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:50.691576 3372 tablet_bootstrap.cc:492] T 4117f23449114eaa814df89f80223e5e P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:50.691650 3481 tablet_bootstrap.cc:654] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:50.691720 3372 ts_tablet_manager.cc:1367] T 4117f23449114eaa814df89f80223e5e P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 21:26:50.692193 3372 raft_consensus.cc:348] T 4117f23449114eaa814df89f80223e5e P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:50.692340 3372 raft_consensus.cc:374] T 4117f23449114eaa814df89f80223e5e P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:50.692413 3372 raft_consensus.cc:729] T 4117f23449114eaa814df89f80223e5e P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:50.693190 3372 consensus_queue.cc:260] T 4117f23449114eaa814df89f80223e5e P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:50.694588 3479 tablet_bootstrap.cc:492] T 883c729aa26f47178f40a17ad3b7e590 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:50.694924 3479 ts_tablet_manager.cc:1367] T 883c729aa26f47178f40a17ad3b7e590 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20240502 21:26:50.696277 3479 raft_consensus.cc:348] T 883c729aa26f47178f40a17ad3b7e590 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:50.696545 3479 raft_consensus.cc:374] T 883c729aa26f47178f40a17ad3b7e590 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:50.696674 3479 raft_consensus.cc:729] T 883c729aa26f47178f40a17ad3b7e590 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:50.697650 3372 ts_tablet_manager.cc:1398] T 4117f23449114eaa814df89f80223e5e P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 21:26:50.697788 3479 consensus_queue.cc:260] T 883c729aa26f47178f40a17ad3b7e590 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:50.699486 3479 ts_tablet_manager.cc:1398] T 883c729aa26f47178f40a17ad3b7e590 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 21:26:50.699766 3479 tablet_bootstrap.cc:492] T 4117f23449114eaa814df89f80223e5e P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:50.700587 3222 tablet_bootstrap.cc:492] T 51551a1967b74094b373cc030e2bf850 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:50.700732 3222 ts_tablet_manager.cc:1367] T 51551a1967b74094b373cc030e2bf850 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.045s user 0.000s sys 0.005s
I20240502 21:26:50.701503 3222 raft_consensus.cc:348] T 51551a1967b74094b373cc030e2bf850 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:50.701690 3222 raft_consensus.cc:374] T 51551a1967b74094b373cc030e2bf850 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:50.701774 3222 raft_consensus.cc:729] T 51551a1967b74094b373cc030e2bf850 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:50.702037 3479 tablet_bootstrap.cc:654] T 4117f23449114eaa814df89f80223e5e P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:50.701983 3222 consensus_queue.cc:260] T 51551a1967b74094b373cc030e2bf850 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:50.703473 3222 ts_tablet_manager.cc:1398] T 51551a1967b74094b373cc030e2bf850 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:26:50.709549 3479 tablet_bootstrap.cc:492] T 4117f23449114eaa814df89f80223e5e P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:50.709743 3479 ts_tablet_manager.cc:1367] T 4117f23449114eaa814df89f80223e5e P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.003s
I20240502 21:26:50.710479 3479 raft_consensus.cc:348] T 4117f23449114eaa814df89f80223e5e P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:50.710755 3479 raft_consensus.cc:374] T 4117f23449114eaa814df89f80223e5e P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:50.710899 3479 raft_consensus.cc:729] T 4117f23449114eaa814df89f80223e5e P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:50.711249 3479 consensus_queue.cc:260] T 4117f23449114eaa814df89f80223e5e P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:50.712463 3479 ts_tablet_manager.cc:1398] T 4117f23449114eaa814df89f80223e5e P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:26:50.729058 3481 tablet_bootstrap.cc:492] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:26:50.729197 3481 ts_tablet_manager.cc:1367] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.071s user 0.006s sys 0.003s
I20240502 21:26:50.729727 3481 raft_consensus.cc:348] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:50.729867 3481 raft_consensus.cc:374] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:50.729975 3481 raft_consensus.cc:729] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:26:50.730180 3481 consensus_queue.cc:260] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:50.730751 3481 ts_tablet_manager.cc:1398] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 21:26:50.731684 2949 raft_consensus.cc:482] T 4117f23449114eaa814df89f80223e5e P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:50.731804 2949 raft_consensus.cc:504] T 4117f23449114eaa814df89f80223e5e P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:50.732460 2949 leader_election.cc:288] T 4117f23449114eaa814df89f80223e5e P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:26:50.732789 31667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4117f23449114eaa814df89f80223e5e" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:50.733042 31667 raft_consensus.cc:2454] T 4117f23449114eaa814df89f80223e5e P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 0.
I20240502 21:26:50.733042 1833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4117f23449114eaa814df89f80223e5e" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:26:50.733247 1833 raft_consensus.cc:2454] T 4117f23449114eaa814df89f80223e5e P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 0.
I20240502 21:26:50.733464 2033 leader_election.cc:302] T 4117f23449114eaa814df89f80223e5e P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 1ade0354c44c4c37ae480c7a94b11e1b; no voters:
I20240502 21:26:50.733841 2949 raft_consensus.cc:2781] T 4117f23449114eaa814df89f80223e5e P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:50.733964 2949 raft_consensus.cc:482] T 4117f23449114eaa814df89f80223e5e P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:50.734058 2949 raft_consensus.cc:3037] T 4117f23449114eaa814df89f80223e5e P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:50.738395 2949 raft_consensus.cc:504] T 4117f23449114eaa814df89f80223e5e P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:50.739324 31667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4117f23449114eaa814df89f80223e5e" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:50.739495 31667 raft_consensus.cc:3037] T 4117f23449114eaa814df89f80223e5e P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:50.742090 31667 raft_consensus.cc:2454] T 4117f23449114eaa814df89f80223e5e P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 1.
I20240502 21:26:50.742710 2033 leader_election.cc:302] T 4117f23449114eaa814df89f80223e5e P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 1ade0354c44c4c37ae480c7a94b11e1b; no voters:
I20240502 21:26:50.742815 31389 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:50.743007 31389 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:50.744110 2949 leader_election.cc:288] T 4117f23449114eaa814df89f80223e5e P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 election: Requested vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:26:50.744158 1833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4117f23449114eaa814df89f80223e5e" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:26:50.744376 1833 raft_consensus.cc:3037] T 4117f23449114eaa814df89f80223e5e P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:50.744460 2949 raft_consensus.cc:2781] T 4117f23449114eaa814df89f80223e5e P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:50.744583 2949 raft_consensus.cc:686] T 4117f23449114eaa814df89f80223e5e P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEADER]: Becoming Leader. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Running, Role: LEADER
I20240502 21:26:50.744827 2949 consensus_queue.cc:237] T 4117f23449114eaa814df89f80223e5e P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:50.746881 1833 raft_consensus.cc:2454] T 4117f23449114eaa814df89f80223e5e P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 1.
I20240502 21:26:50.746902 31389 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet aca362d1a6894071b195dd9fb7594b1f
I20240502 21:26:50.748147 31647 tablet_service.cc:1514] Processing DeleteTablet for tablet aca362d1a6894071b195dd9fb7594b1f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:50 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:50.751899 2210 tablet_service.cc:1514] Processing DeleteTablet for tablet aca362d1a6894071b195dd9fb7594b1f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:50 UTC) from {username='slave'} at 127.0.0.1:36984
I20240502 21:26:50.752251 2075 tablet_service.cc:1514] Processing DeleteTablet for tablet aca362d1a6894071b195dd9fb7594b1f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:50 UTC) from {username='slave'} at 127.0.0.1:36740
I20240502 21:26:50.753705 3491 tablet_replica.cc:330] T aca362d1a6894071b195dd9fb7594b1f P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:50.753981 3491 raft_consensus.cc:2229] T aca362d1a6894071b195dd9fb7594b1f P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:50.757319 3491 raft_consensus.cc:2258] T aca362d1a6894071b195dd9fb7594b1f P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:50.758338 3491 ts_tablet_manager.cc:1875] T aca362d1a6894071b195dd9fb7594b1f P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:50.762229 31389 catalog_manager.cc:5484] T 4117f23449114eaa814df89f80223e5e P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135). New cstate: current_term: 1 leader_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:50.777150 3491 ts_tablet_manager.cc:1888] T aca362d1a6894071b195dd9fb7594b1f P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:50.777323 3491 log.cc:1196] T aca362d1a6894071b195dd9fb7594b1f P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/aca362d1a6894071b195dd9fb7594b1f
I20240502 21:26:50.777809 3491 ts_tablet_manager.cc:1909] T aca362d1a6894071b195dd9fb7594b1f P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:26:50.778808 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet aca362d1a6894071b195dd9fb7594b1f (table rebalancer_dynamic_table_new_name [id=2faf275486e74ad19118c3628524b2e6]) successfully deleted
I20240502 21:26:50.781549 3493 tablet_replica.cc:330] T aca362d1a6894071b195dd9fb7594b1f P 1ade0354c44c4c37ae480c7a94b11e1b: stopping tablet replica
I20240502 21:26:50.784494 3392 raft_consensus.cc:482] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:50.784696 3392 raft_consensus.cc:504] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:50.785235 3492 tablet_replica.cc:330] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d: stopping tablet replica
I20240502 21:26:50.784842 3493 raft_consensus.cc:2229] T aca362d1a6894071b195dd9fb7594b1f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:50.785324 3392 leader_election.cc:288] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:50.785578 3493 raft_consensus.cc:2258] T aca362d1a6894071b195dd9fb7594b1f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:50.785915 3492 raft_consensus.cc:2229] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:50.786182 31667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51551a1967b74094b373cc030e2bf850" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:50.786296 3492 raft_consensus.cc:2258] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:50.786391 31667 raft_consensus.cc:2454] T 51551a1967b74094b373cc030e2bf850 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 0.
I20240502 21:26:50.786784 1902 leader_election.cc:302] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:26:50.787092 3392 raft_consensus.cc:2781] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:50.787190 3392 raft_consensus.cc:482] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:50.787273 3392 raft_consensus.cc:3037] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:50.787688 2230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51551a1967b74094b373cc030e2bf850" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:26:50.787861 2230 raft_consensus.cc:2454] T 51551a1967b74094b373cc030e2bf850 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 0.
I20240502 21:26:50.789106 3493 ts_tablet_manager.cc:1875] T aca362d1a6894071b195dd9fb7594b1f P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:50.789767 3392 raft_consensus.cc:504] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:50.790928 3492 ts_tablet_manager.cc:1875] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:50.791651 31667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51551a1967b74094b373cc030e2bf850" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:50.791824 31667 raft_consensus.cc:3037] T 51551a1967b74094b373cc030e2bf850 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:50.794466 31667 raft_consensus.cc:2454] T 51551a1967b74094b373cc030e2bf850 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 1.
I20240502 21:26:50.795209 2230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51551a1967b74094b373cc030e2bf850" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:26:50.795365 2230 raft_consensus.cc:3037] T 51551a1967b74094b373cc030e2bf850 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:50.796764 1902 leader_election.cc:302] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:26:50.797305 3392 leader_election.cc:288] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 election: Requested vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:50.797824 3392 raft_consensus.cc:2781] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:50.798002 3392 raft_consensus.cc:686] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf [term 1 LEADER]: Becoming Leader. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Running, Role: LEADER
I20240502 21:26:50.798256 3392 consensus_queue.cc:237] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:50.797696 2230 raft_consensus.cc:2454] T 51551a1967b74094b373cc030e2bf850 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 1.
I20240502 21:26:50.809511 3492 ts_tablet_manager.cc:1888] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:50.810218 3493 ts_tablet_manager.cc:1888] T aca362d1a6894071b195dd9fb7594b1f P 1ade0354c44c4c37ae480c7a94b11e1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:50.812037 31389 catalog_manager.cc:5484] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dec645b473f40578e726f3e50e219bf (127.24.226.134). New cstate: current_term: 1 leader_uuid: "9dec645b473f40578e726f3e50e219bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:50.812930 3493 log.cc:1196] T aca362d1a6894071b195dd9fb7594b1f P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal/wals/aca362d1a6894071b195dd9fb7594b1f
I20240502 21:26:50.813367 3493 ts_tablet_manager.cc:1909] T aca362d1a6894071b195dd9fb7594b1f P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting consensus metadata
I20240502 21:26:50.813508 3492 log.cc:1196] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal/wals/aca362d1a6894071b195dd9fb7594b1f
I20240502 21:26:50.814050 3492 ts_tablet_manager.cc:1909] T aca362d1a6894071b195dd9fb7594b1f P d1406e92aec84ac585304e57bd37a40d: Deleting consensus metadata
I20240502 21:26:50.815810 31381 catalog_manager.cc:4830] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): tablet aca362d1a6894071b195dd9fb7594b1f (table rebalancer_dynamic_table_new_name [id=2faf275486e74ad19118c3628524b2e6]) successfully deleted
I20240502 21:26:50.816126 31381 catalog_manager.cc:4830] TS d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): tablet aca362d1a6894071b195dd9fb7594b1f (table rebalancer_dynamic_table_new_name [id=2faf275486e74ad19118c3628524b2e6]) successfully deleted
I20240502 21:26:50.826918 31925 consensus_queue.cc:237] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } }
I20240502 21:26:50.828909 1833 raft_consensus.cc:1261] T dab4f06cc78946dc962267cb2aae04f7 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:50.829289 2230 raft_consensus.cc:1261] T dab4f06cc78946dc962267cb2aae04f7 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:50.829403 1884 consensus_queue.cc:1035] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:26:50.829699 31862 consensus_peers.cc:485] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 -> Peer 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): Couldn't send request to peer 1ade0354c44c4c37ae480c7a94b11e1b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dab4f06cc78946dc962267cb2aae04f7. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:50.830107 1884 consensus_queue.cc:1035] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:50.831833 3501 raft_consensus.cc:2932] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } } }
I20240502 21:26:50.832553 1833 raft_consensus.cc:2932] T dab4f06cc78946dc962267cb2aae04f7 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } } }
I20240502 21:26:50.836611 31389 catalog_manager.cc:5484] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: config changed from index -1 to 2, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:50.843420 2230 raft_consensus.cc:2932] T dab4f06cc78946dc962267cb2aae04f7 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } } }
I20240502 21:26:50.849828 31389 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:26:50.857966 1812 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca6669bb60a4b7eb7b01acb8fc64800 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2bf308935f084e238190eddf14154196]), partition=
I20240502 21:26:50.858340 1812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ca6669bb60a4b7eb7b01acb8fc64800. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:50.858651 31647 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca6669bb60a4b7eb7b01acb8fc64800 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2bf308935f084e238190eddf14154196]), partition=
I20240502 21:26:50.858969 31647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ca6669bb60a4b7eb7b01acb8fc64800. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:50.860953 2210 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca6669bb60a4b7eb7b01acb8fc64800 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2bf308935f084e238190eddf14154196]), partition=
I20240502 21:26:50.861238 2210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ca6669bb60a4b7eb7b01acb8fc64800. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:50.864672 3408 tablet_bootstrap.cc:492] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:50.866007 3408 tablet_bootstrap.cc:654] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:50.867384 31925 consensus_queue.cc:237] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:26:50.869731 2230 raft_consensus.cc:1261] T 62f083e9641042c990efcfb1c4104de5 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 21:26:50.869760 31860 consensus_peers.cc:485] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62f083e9641042c990efcfb1c4104de5. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:50.869863 31797 raft_consensus.cc:1261] T 62f083e9641042c990efcfb1c4104de5 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:26:50.870308 1884 consensus_queue.cc:1035] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:26:50.870687 3408 tablet_bootstrap.cc:492] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:50.870831 3408 ts_tablet_manager.cc:1367] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 21:26:50.870813 1884 consensus_queue.cc:1035] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:26:50.871546 3408 raft_consensus.cc:348] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:50.871757 3408 raft_consensus.cc:374] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:50.871836 3408 raft_consensus.cc:729] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:50.872822 3408 consensus_queue.cc:260] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:50.873234 3501 raft_consensus.cc:2932] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:50.873864 2228 raft_consensus.cc:2932] T 62f083e9641042c990efcfb1c4104de5 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:50.874267 3408 ts_tablet_manager.cc:1398] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:50.876317 3222 tablet_bootstrap.cc:492] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:50.877822 3222 tablet_bootstrap.cc:654] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:50.878893 3479 tablet_bootstrap.cc:492] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:50.880043 3479 tablet_bootstrap.cc:654] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:50.882997 31391 catalog_manager.cc:5484] T 62f083e9641042c990efcfb1c4104de5 P d1406e92aec84ac585304e57bd37a40d reported cstate change: config changed from index 6 to 7, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:50.887933 3222 tablet_bootstrap.cc:492] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:50.888159 3222 ts_tablet_manager.cc:1367] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240502 21:26:50.888999 3222 raft_consensus.cc:348] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:50.889349 3222 raft_consensus.cc:374] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:50.889508 3222 raft_consensus.cc:729] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:50.889940 3222 consensus_queue.cc:260] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:50.890563 3222 ts_tablet_manager.cc:1398] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:50.890858 2891 raft_consensus.cc:482] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:50.891006 2891 raft_consensus.cc:504] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:50.891542 2891 leader_election.cc:288] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:50.892113 2097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "883c729aa26f47178f40a17ad3b7e590" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:50.892313 2097 raft_consensus.cc:2454] T 883c729aa26f47178f40a17ad3b7e590 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:26:50.892887 2163 leader_election.cc:302] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:50.895620 1833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "883c729aa26f47178f40a17ad3b7e590" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:26:50.895869 1833 raft_consensus.cc:2454] T 883c729aa26f47178f40a17ad3b7e590 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:26:50.896612 2891 raft_consensus.cc:2781] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:50.896740 2891 raft_consensus.cc:482] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:50.896875 2891 raft_consensus.cc:3037] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:50.896641 31797 raft_consensus.cc:2932] T 62f083e9641042c990efcfb1c4104de5 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:50.899909 2891 raft_consensus.cc:504] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:50.901134 1833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "883c729aa26f47178f40a17ad3b7e590" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:26:50.901332 1833 raft_consensus.cc:3037] T 883c729aa26f47178f40a17ad3b7e590 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:50.903985 1833 raft_consensus.cc:2454] T 883c729aa26f47178f40a17ad3b7e590 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 1.
I20240502 21:26:50.904842 2163 leader_election.cc:302] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:50.905794 2097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "883c729aa26f47178f40a17ad3b7e590" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:26:50.905973 2097 raft_consensus.cc:3037] T 883c729aa26f47178f40a17ad3b7e590 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:50.906845 3479 tablet_bootstrap.cc:492] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:50.906960 3479 ts_tablet_manager.cc:1367] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.010s
I20240502 21:26:50.908130 2097 raft_consensus.cc:2454] T 883c729aa26f47178f40a17ad3b7e590 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 1.
I20240502 21:26:50.908243 3479 raft_consensus.cc:348] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:50.908411 3479 raft_consensus.cc:374] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:50.908555 3479 raft_consensus.cc:729] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:50.908625 2891 leader_election.cc:288] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Requested vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:50.908898 2891 raft_consensus.cc:2781] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:50.909152 2891 raft_consensus.cc:686] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Becoming Leader. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Running, Role: LEADER
I20240502 21:26:50.909375 2891 consensus_queue.cc:237] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:50.909418 3479 consensus_queue.cc:260] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:50.910259 3479 ts_tablet_manager.cc:1398] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:26:50.925594 31797 consensus_queue.cc:237] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } }
I20240502 21:26:50.927155 2429 raft_consensus.cc:482] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:50.927320 2429 raft_consensus.cc:504] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:50.927968 2429 leader_election.cc:288] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:50.929133 31667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca6669bb60a4b7eb7b01acb8fc64800" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:50.929354 31667 raft_consensus.cc:2454] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:26:50.929769 2164 leader_election.cc:302] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:50.930325 2429 raft_consensus.cc:2781] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:50.930423 2429 raft_consensus.cc:482] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:50.930510 2429 raft_consensus.cc:3037] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:50.931818 1833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca6669bb60a4b7eb7b01acb8fc64800" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:26:50.932053 1833 raft_consensus.cc:2454] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:26:50.933746 2429 raft_consensus.cc:504] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
W20240502 21:26:50.934439 31732 consensus_peers.cc:485] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c -> Peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): Couldn't send request to peer 1e04157b98dd4fad98858c290b73c3db. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a1fd14de204c40d98027ecb329ff79f3. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:50.934746 2429 leader_election.cc:288] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Requested vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:50.934890 31536 raft_consensus.cc:1261] T a1fd14de204c40d98027ecb329ff79f3 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:50.935372 1833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca6669bb60a4b7eb7b01acb8fc64800" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:26:50.935573 1833 raft_consensus.cc:3037] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:50.935650 3520 consensus_queue.cc:1035] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:50.936633 31667 raft_consensus.cc:1261] T a1fd14de204c40d98027ecb329ff79f3 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:50.938218 1833 raft_consensus.cc:2454] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 1.
I20240502 21:26:50.938028 31395 catalog_manager.cc:5484] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d reported cstate change: term changed from 0 to 1, leader changed from <none> to d1406e92aec84ac585304e57bd37a40d (127.24.226.136). New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:50.938570 2163 leader_election.cc:302] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:50.938807 2429 raft_consensus.cc:2781] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:50.938980 2429 raft_consensus.cc:686] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Becoming Leader. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Running, Role: LEADER
I20240502 21:26:50.939240 2429 consensus_queue.cc:237] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:50.944648 31667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca6669bb60a4b7eb7b01acb8fc64800" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:50.944937 31667 raft_consensus.cc:3037] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:50.948027 3520 consensus_queue.cc:1035] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:50.952906 31667 raft_consensus.cc:2454] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 1.
I20240502 21:26:50.954388 3033 raft_consensus.cc:2932] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } } }
I20240502 21:26:50.955412 31395 catalog_manager.cc:5484] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d reported cstate change: term changed from 0 to 1, leader changed from <none> to d1406e92aec84ac585304e57bd37a40d (127.24.226.136). New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:50.957237 31535 raft_consensus.cc:2932] T a1fd14de204c40d98027ecb329ff79f3 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } } }
I20240502 21:26:50.959401 31666 raft_consensus.cc:2932] T a1fd14de204c40d98027ecb329ff79f3 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } } }
I20240502 21:26:50.965833 31391 catalog_manager.cc:5484] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index -1 to 4, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:50.981590 31797 consensus_queue.cc:237] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } }
W20240502 21:26:50.984028 31732 consensus_peers.cc:485] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c -> Peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): Couldn't send request to peer d1406e92aec84ac585304e57bd37a40d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 53b7eec870c44999b48f0bf744b2ffcd. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:50.984294 31536 raft_consensus.cc:1261] T 53b7eec870c44999b48f0bf744b2ffcd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:50.984313 31925 raft_consensus.cc:1261] T 53b7eec870c44999b48f0bf744b2ffcd P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:50.984916 3520 consensus_queue.cc:1035] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:50.985146 3033 consensus_queue.cc:1035] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:50.987793 3520 raft_consensus.cc:2932] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } } }
I20240502 21:26:50.988328 31536 raft_consensus.cc:2932] T 53b7eec870c44999b48f0bf744b2ffcd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } } }
I20240502 21:26:50.993288 31395 catalog_manager.cc:5484] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index -1 to 4, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:51.005908 31925 raft_consensus.cc:2932] T 53b7eec870c44999b48f0bf744b2ffcd P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } } }
I20240502 21:26:51.020486 31925 consensus_queue.cc:237] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } }
I20240502 21:26:51.023121 31536 raft_consensus.cc:1261] T 88189733d9a14560971dbeb71543073f P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:26:51.023489 31862 consensus_peers.cc:485] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 -> Peer 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): Couldn't send request to peer 1ade0354c44c4c37ae480c7a94b11e1b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88189733d9a14560971dbeb71543073f. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:51.023614 31797 raft_consensus.cc:1261] T 88189733d9a14560971dbeb71543073f P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:51.023931 3504 consensus_queue.cc:1035] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:51.024631 3504 consensus_queue.cc:1035] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:51.026882 3504 raft_consensus.cc:2932] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } } }
I20240502 21:26:51.027338 31536 raft_consensus.cc:2932] T 88189733d9a14560971dbeb71543073f P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } } }
I20240502 21:26:51.029311 31797 raft_consensus.cc:2932] T 88189733d9a14560971dbeb71543073f P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } } }
I20240502 21:26:51.035125 31395 catalog_manager.cc:5484] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:51.054433 31536 consensus_queue.cc:237] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:26:51.056582 31797 raft_consensus.cc:1261] T 759b7529d7d84baab1a4622d34f803bf P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:51.056596 31924 raft_consensus.cc:1261] T 759b7529d7d84baab1a4622d34f803bf P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:26:51.057233 31469 consensus_peers.cc:485] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 759b7529d7d84baab1a4622d34f803bf. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:51.057225 3207 consensus_queue.cc:1035] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:51.057513 2631 consensus_queue.cc:1035] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:51.060395 3553 raft_consensus.cc:2932] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:51.061127 31797 raft_consensus.cc:2932] T 759b7529d7d84baab1a4622d34f803bf P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:51.061053 31924 raft_consensus.cc:2932] T 759b7529d7d84baab1a4622d34f803bf P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:51.066174 31391 catalog_manager.cc:5484] T 759b7529d7d84baab1a4622d34f803bf P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:51.070768 31389 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:26:51.076498 31667 raft_consensus.cc:1261] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:51.076709 1833 raft_consensus.cc:1261] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:51.077273 2891 consensus_queue.cc:1035] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:51.077535 2429 consensus_queue.cc:1035] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:51.081008 31797 consensus_queue.cc:237] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } }
W20240502 21:26:51.084292 31732 consensus_peers.cc:485] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c -> Peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): Couldn't send request to peer d1406e92aec84ac585304e57bd37a40d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a54d2aad63c4f3182992f00b8564acf. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:51.084583 31666 raft_consensus.cc:1261] T 5a54d2aad63c4f3182992f00b8564acf P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:51.085202 31924 raft_consensus.cc:1261] T 5a54d2aad63c4f3182992f00b8564acf P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:51.085567 3517 consensus_queue.cc:1035] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:51.085856 3520 consensus_queue.cc:1035] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:51.094481 3033 raft_consensus.cc:2932] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } } }
I20240502 21:26:51.099138 3568 tablet.cc:1722] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:51.099009 31924 raft_consensus.cc:2932] T 5a54d2aad63c4f3182992f00b8564acf P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } } }
I20240502 21:26:51.099951 3244 tablet.cc:1722] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 05f4fac41c4f4959b9b316cb112d0ada: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:51.100281 3568 tablet.cc:1620] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d: MemRowSet was empty: no flush needed.
I20240502 21:26:51.100385 3244 tablet.cc:1620] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 05f4fac41c4f4959b9b316cb112d0ada: MemRowSet was empty: no flush needed.
I20240502 21:26:51.101428 31389 catalog_manager.cc:5576] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645) does not have the latest schema for tablet 1ca6669bb60a4b7eb7b01acb8fc64800 (table rebalancer_dynamic_table [id=2bf308935f084e238190eddf14154196]). Expected version 1 got 0
I20240502 21:26:51.101271 31666 raft_consensus.cc:2932] T 5a54d2aad63c4f3182992f00b8564acf P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } } }
I20240502 21:26:51.107697 31395 catalog_manager.cc:5484] T 5a54d2aad63c4f3182992f00b8564acf P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: config changed from index -1 to 4, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } } }
I20240502 21:26:51.116681 31389 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=2bf308935f084e238190eddf14154196] alter complete (version 1)
I20240502 21:26:51.131068 31536 consensus_queue.cc:237] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } }
W20240502 21:26:51.135018 31472 consensus_peers.cc:485] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f -> Peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): Couldn't send request to peer 2d50f33557944bb480ea32199d9ee4f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5bc6c7211d5f4f73b2a050b9ec78c6b1. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:51.135098 1834 raft_consensus.cc:1261] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:51.135761 2097 raft_consensus.cc:1261] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:51.135797 3207 consensus_queue.cc:1035] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:51.136545 3207 consensus_queue.cc:1035] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:51.139760 2631 raft_consensus.cc:2932] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } } }
I20240502 21:26:51.140465 2097 raft_consensus.cc:2932] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } } }
I20240502 21:26:51.147169 31395 catalog_manager.cc:5484] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: config changed from index -1 to 2, NON_VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) added. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } } }
I20240502 21:26:51.169854 31395 catalog_manager.cc:5576] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863) does not have the latest schema for tablet 1ca6669bb60a4b7eb7b01acb8fc64800 (table rebalancer_dynamic_table [id=2bf308935f084e238190eddf14154196]). Expected version 1 got 0
I20240502 21:26:51.173231 1834 raft_consensus.cc:2932] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } } }
I20240502 21:26:51.190786 31395 catalog_manager.cc:5576] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863) does not have the latest schema for tablet 1ca6669bb60a4b7eb7b01acb8fc64800 (table rebalancer_dynamic_table [id=2bf308935f084e238190eddf14154196]). Expected version 1 got 0
I20240502 21:26:51.193086 3581 tablet.cc:1722] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:51.193750 3581 tablet.cc:1620] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 1e04157b98dd4fad98858c290b73c3db: MemRowSet was empty: no flush needed.
I20240502 21:26:51.214160 31797 consensus_queue.cc:237] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } }
I20240502 21:26:51.216537 31666 raft_consensus.cc:1261] T c5c344f8f84f4a73b853b4f908b93964 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:51.217151 3033 consensus_queue.cc:1035] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:51.218154 31536 raft_consensus.cc:1261] T c5c344f8f84f4a73b853b4f908b93964 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:51.218642 3567 consensus_queue.cc:1035] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:51.220574 3033 raft_consensus.cc:2932] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } } }
I20240502 21:26:51.221164 31666 raft_consensus.cc:2932] T c5c344f8f84f4a73b853b4f908b93964 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } } }
I20240502 21:26:51.224298 31395 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:26:51.226213 2949 consensus_queue.cc:1035] T 4117f23449114eaa814df89f80223e5e P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:51.230226 31389 catalog_manager.cc:5484] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:51.239323 3568 tablet.cc:1722] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:51.239794 3568 tablet.cc:1620] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d: MemRowSet was empty: no flush needed.
I20240502 21:26:51.243870 3581 tablet.cc:1722] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:51.244352 3581 tablet.cc:1620] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 1e04157b98dd4fad98858c290b73c3db: MemRowSet was empty: no flush needed.
I20240502 21:26:51.249388 31535 raft_consensus.cc:2932] T c5c344f8f84f4a73b853b4f908b93964 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } } }
I20240502 21:26:51.251008 3244 tablet.cc:1722] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 05f4fac41c4f4959b9b316cb112d0ada: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:51.251535 3244 tablet.cc:1620] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 05f4fac41c4f4959b9b316cb112d0ada: MemRowSet was empty: no flush needed.
I20240502 21:26:51.260154 3489 consensus_queue.cc:1035] T 4117f23449114eaa814df89f80223e5e P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:51.264600 31395 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=2bf308935f084e238190eddf14154196] alter complete (version 2)
W20240502 21:26:51.269428 31732 consensus_peers.cc:485] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c -> Peer 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): Couldn't send request to peer 1ade0354c44c4c37ae480c7a94b11e1b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5c344f8f84f4a73b853b4f908b93964. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:51.281247 31925 consensus_queue.cc:237] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:26:51.283270 31667 raft_consensus.cc:1261] T ae56d18c629a4e5eb50911c89de4227c P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:26:51.283543 31860 consensus_peers.cc:485] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae56d18c629a4e5eb50911c89de4227c. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:51.283735 31797 raft_consensus.cc:1261] T ae56d18c629a4e5eb50911c89de4227c P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:51.283810 1884 consensus_queue.cc:1035] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:51.284238 3501 consensus_queue.cc:1035] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:51.286209 3504 raft_consensus.cc:2932] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:51.286573 3392 consensus_queue.cc:1035] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:51.286720 31666 raft_consensus.cc:2932] T ae56d18c629a4e5eb50911c89de4227c P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:51.292547 31389 catalog_manager.cc:5484] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:51.293985 31797 raft_consensus.cc:2932] T ae56d18c629a4e5eb50911c89de4227c P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:51.305627 3392 consensus_queue.cc:1035] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:51.315461 31535 consensus_queue.cc:237] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } }
I20240502 21:26:51.318046 1966 raft_consensus.cc:1261] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:26:51.318552 31666 raft_consensus.cc:1261] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:26:51.318511 3578 consensus_queue.cc:1035] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:26:51.319073 3553 consensus_queue.cc:1035] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 21:26:51.320582 31471 consensus_peers.cc:485] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f -> Peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): Couldn't send request to peer d1406e92aec84ac585304e57bd37a40d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 55fc18e4cc98446ca50c1dd2b6a9a89d. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:51.321800 3553 raft_consensus.cc:2932] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } } }
I20240502 21:26:51.322222 1966 raft_consensus.cc:2932] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } } }
I20240502 21:26:51.325426 31666 raft_consensus.cc:2932] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } } }
I20240502 21:26:51.328646 31390 catalog_manager.cc:5484] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 9dec645b473f40578e726f3e50e219bf reported cstate change: config changed from index 6 to 7, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } } }
I20240502 21:26:51.347103 31795 consensus_queue.cc:237] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } }
I20240502 21:26:51.351063 3616 ts_tablet_manager.cc:902] T dab4f06cc78946dc962267cb2aae04f7 P 1ade0354c44c4c37ae480c7a94b11e1b: Initiating tablet copy from peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:51.353533 2230 raft_consensus.cc:1261] T 5b1dcae186f542cea92c9191fa8ef951 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:26:51.354179 3033 consensus_queue.cc:1035] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 21:26:51.354604 31732 consensus_peers.cc:485] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c -> Peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): Couldn't send request to peer 1e04157b98dd4fad98858c290b73c3db. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5b1dcae186f542cea92c9191fa8ef951. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:51.356069 31925 raft_consensus.cc:1261] T 5b1dcae186f542cea92c9191fa8ef951 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:26:51.356593 3520 consensus_queue.cc:1035] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:26:51.359061 3520 raft_consensus.cc:2932] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } } }
I20240502 21:26:51.363116 2230 raft_consensus.cc:2932] T 5b1dcae186f542cea92c9191fa8ef951 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } } }
I20240502 21:26:51.365103 3616 tablet_copy_client.cc:308] T dab4f06cc78946dc962267cb2aae04f7 P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.132:35933
I20240502 21:26:51.365695 31947 tablet_copy_service.cc:140] P 2d50f33557944bb480ea32199d9ee4f1: Received BeginTabletCopySession request for tablet dab4f06cc78946dc962267cb2aae04f7 from peer 1ade0354c44c4c37ae480c7a94b11e1b ({username='slave'} at 127.24.226.135:33163)
I20240502 21:26:51.365841 31947 tablet_copy_service.cc:161] P 2d50f33557944bb480ea32199d9ee4f1: Beginning new tablet copy session on tablet dab4f06cc78946dc962267cb2aae04f7 from peer 1ade0354c44c4c37ae480c7a94b11e1b at {username='slave'} at 127.24.226.135:33163: session id = 1ade0354c44c4c37ae480c7a94b11e1b-dab4f06cc78946dc962267cb2aae04f7
I20240502 21:26:51.368353 31947 tablet_copy_source_session.cc:205] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:51.368280 31395 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:26:51.373749 31925 raft_consensus.cc:2932] T 5b1dcae186f542cea92c9191fa8ef951 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } } }
I20240502 21:26:51.374109 31390 catalog_manager.cc:5484] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index 6 to 7, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:51.376401 3616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dab4f06cc78946dc962267cb2aae04f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:51.380970 3568 tablet.cc:1722] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:51.384727 3581 tablet.cc:1722] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:51.390517 3244 tablet.cc:1722] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 05f4fac41c4f4959b9b316cb112d0ada: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:51.396291 3616 tablet_copy_client.cc:783] T dab4f06cc78946dc962267cb2aae04f7 P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:51.396569 3616 tablet_copy_client.cc:647] T dab4f06cc78946dc962267cb2aae04f7 P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:51.398602 3616 tablet_copy_client.cc:523] T dab4f06cc78946dc962267cb2aae04f7 P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:51.401119 31389 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table_new_name [id=2bf308935f084e238190eddf14154196] alter complete (version 3)
I20240502 21:26:51.405287 3616 tablet_bootstrap.cc:492] T dab4f06cc78946dc962267cb2aae04f7 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:51.411235 3616 tablet_bootstrap.cc:492] T dab4f06cc78946dc962267cb2aae04f7 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:51.411639 3616 tablet_bootstrap.cc:492] T dab4f06cc78946dc962267cb2aae04f7 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap complete.
I20240502 21:26:51.411871 3616 ts_tablet_manager.cc:1367] T dab4f06cc78946dc962267cb2aae04f7 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20240502 21:26:51.412243 1966 consensus_queue.cc:237] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } }
I20240502 21:26:51.412617 3616 raft_consensus.cc:348] T dab4f06cc78946dc962267cb2aae04f7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } }
I20240502 21:26:51.412819 3616 raft_consensus.cc:729] T dab4f06cc78946dc962267cb2aae04f7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: LEARNER
I20240502 21:26:51.413023 3616 consensus_queue.cc:260] T dab4f06cc78946dc962267cb2aae04f7 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } }
I20240502 21:26:51.413777 3616 ts_tablet_manager.cc:1398] T dab4f06cc78946dc962267cb2aae04f7 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:26:51.414412 31947 tablet_copy_service.cc:342] P 2d50f33557944bb480ea32199d9ee4f1: Request end of tablet copy session 1ade0354c44c4c37ae480c7a94b11e1b-dab4f06cc78946dc962267cb2aae04f7 received from {username='slave'} at 127.24.226.135:33163
I20240502 21:26:51.414561 31947 tablet_copy_service.cc:433] P 2d50f33557944bb480ea32199d9ee4f1: ending tablet copy session 1ade0354c44c4c37ae480c7a94b11e1b-dab4f06cc78946dc962267cb2aae04f7 on tablet dab4f06cc78946dc962267cb2aae04f7 with peer 1ade0354c44c4c37ae480c7a94b11e1b
I20240502 21:26:51.415619 1832 raft_consensus.cc:1261] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:51.415784 2097 raft_consensus.cc:1261] T acb3bdc518b84b5abc362cc4e12027c3 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:51.416211 3610 consensus_queue.cc:1035] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:51.416694 3403 consensus_queue.cc:1035] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:26:51.417137 1902 consensus_peers.cc:485] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf -> Peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Couldn't send request to peer a44f41b0b6644fb6aa29476aed8c964f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: acb3bdc518b84b5abc362cc4e12027c3. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:51.419994 3403 raft_consensus.cc:2932] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } } }
I20240502 21:26:51.420377 1832 raft_consensus.cc:2932] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } } }
I20240502 21:26:51.426128 2097 raft_consensus.cc:2932] T acb3bdc518b84b5abc362cc4e12027c3 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } } }
I20240502 21:26:51.426587 31395 catalog_manager.cc:5484] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db reported cstate change: config changed from index -1 to 2, NON_VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) added. New cstate: current_term: 1 leader_uuid: "9dec645b473f40578e726f3e50e219bf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } } }
I20240502 21:26:51.426864 2429 consensus_queue.cc:1035] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:51.442270 2891 consensus_queue.cc:1035] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:51.457444 3644 ts_tablet_manager.cc:902] T 62f083e9641042c990efcfb1c4104de5 P 9dec645b473f40578e726f3e50e219bf: Initiating tablet copy from peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:51.458837 3643 ts_tablet_manager.cc:902] T 759b7529d7d84baab1a4622d34f803bf P 9dec645b473f40578e726f3e50e219bf: Initiating tablet copy from peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:51.462582 3651 ts_tablet_manager.cc:902] T a1fd14de204c40d98027ecb329ff79f3 P 1e04157b98dd4fad98858c290b73c3db: Initiating tablet copy from peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:51.469931 3656 ts_tablet_manager.cc:902] T 53b7eec870c44999b48f0bf744b2ffcd P d1406e92aec84ac585304e57bd37a40d: Initiating tablet copy from peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:51.470785 31390 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_extra_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:51.480407 3643 tablet_copy_client.cc:308] T 759b7529d7d84baab1a4622d34f803bf P 9dec645b473f40578e726f3e50e219bf: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.129:41039
I20240502 21:26:51.481316 31556 tablet_copy_service.cc:140] P a44f41b0b6644fb6aa29476aed8c964f: Received BeginTabletCopySession request for tablet 759b7529d7d84baab1a4622d34f803bf from peer 9dec645b473f40578e726f3e50e219bf ({username='slave'} at 127.24.226.134:40609)
I20240502 21:26:51.481575 31556 tablet_copy_service.cc:161] P a44f41b0b6644fb6aa29476aed8c964f: Beginning new tablet copy session on tablet 759b7529d7d84baab1a4622d34f803bf from peer 9dec645b473f40578e726f3e50e219bf at {username='slave'} at 127.24.226.134:40609: session id = 9dec645b473f40578e726f3e50e219bf-759b7529d7d84baab1a4622d34f803bf
I20240502 21:26:51.482781 1833 consensus_queue.cc:237] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:26:51.484943 31556 tablet_copy_source_session.cc:205] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:51.486140 2210 tablet_service.cc:1467] Processing CreateTablet for tablet 2e4ed0121a404869b6fa33b14d3c49f8 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=a35acd1fa3294955995e9c2c846a5de0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:51.486411 31905 tablet_service.cc:1467] Processing CreateTablet for tablet 2e4ed0121a404869b6fa33b14d3c49f8 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=a35acd1fa3294955995e9c2c846a5de0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:51.486620 2210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e4ed0121a404869b6fa33b14d3c49f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:51.486884 31905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e4ed0121a404869b6fa33b14d3c49f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:51.487540 31775 tablet_service.cc:1467] Processing CreateTablet for tablet 2c737c49a32b4f3ea5ef735fa254d48e (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=a35acd1fa3294955995e9c2c846a5de0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:51.488003 31775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c737c49a32b4f3ea5ef735fa254d48e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:51.487943 1811 tablet_service.cc:1467] Processing CreateTablet for tablet 2c737c49a32b4f3ea5ef735fa254d48e (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=a35acd1fa3294955995e9c2c846a5de0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:51.488061 1812 tablet_service.cc:1467] Processing CreateTablet for tablet 2e4ed0121a404869b6fa33b14d3c49f8 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=a35acd1fa3294955995e9c2c846a5de0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:51.489243 1811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c737c49a32b4f3ea5ef735fa254d48e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:51.489671 2075 tablet_service.cc:1467] Processing CreateTablet for tablet 2c737c49a32b4f3ea5ef735fa254d48e (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=a35acd1fa3294955995e9c2c846a5de0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:51.490070 2075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c737c49a32b4f3ea5ef735fa254d48e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:51.492712 1812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e4ed0121a404869b6fa33b14d3c49f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:51.500962 31516 tablet_service.cc:1467] Processing CreateTablet for tablet 3e6657723fe84e749dea768de44638e7 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=a35acd1fa3294955995e9c2c846a5de0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:51.501626 31516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e6657723fe84e749dea768de44638e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:51.502769 3664 tablet_bootstrap.cc:492] T 2e4ed0121a404869b6fa33b14d3c49f8 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:51.504983 31395 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:51.505085 2076 tablet_service.cc:1467] Processing CreateTablet for tablet 3e6657723fe84e749dea768de44638e7 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=a35acd1fa3294955995e9c2c846a5de0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:51.505282 31395 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:51.505564 2076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e6657723fe84e749dea768de44638e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:51.506773 1814 tablet_service.cc:1467] Processing CreateTablet for tablet 3e6657723fe84e749dea768de44638e7 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=a35acd1fa3294955995e9c2c846a5de0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:51.508119 1814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e6657723fe84e749dea768de44638e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:51.509722 3664 tablet_bootstrap.cc:654] T 2e4ed0121a404869b6fa33b14d3c49f8 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:51.510838 31395 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 1ca6669bb60a4b7eb7b01acb8fc64800
I20240502 21:26:51.512181 31647 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ca6669bb60a4b7eb7b01acb8fc64800 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:51 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:51.512967 2208 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ca6669bb60a4b7eb7b01acb8fc64800 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:51 UTC) from {username='slave'} at 127.0.0.1:36984
I20240502 21:26:51.515321 1813 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ca6669bb60a4b7eb7b01acb8fc64800 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:51 UTC) from {username='slave'} at 127.0.0.1:47510
I20240502 21:26:51.518380 31666 raft_consensus.cc:1261] T b9342dbe0497499ba4c238f2ed033293 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:51.519245 3529 consensus_queue.cc:1035] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 21:26:51.520159 1767 consensus_peers.cc:485] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b9342dbe0497499ba4c238f2ed033293. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:51.527158 31795 raft_consensus.cc:1261] T b9342dbe0497499ba4c238f2ed033293 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:51.528401 3529 consensus_queue.cc:1035] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:51.535879 3643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 759b7529d7d84baab1a4622d34f803bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:51.542477 3672 tablet_bootstrap.cc:492] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:51.547937 3672 tablet_bootstrap.cc:654] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:51.556192 3656 tablet_copy_client.cc:308] T 53b7eec870c44999b48f0bf744b2ffcd P d1406e92aec84ac585304e57bd37a40d: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.131:40807
I20240502 21:26:51.556932 31817 tablet_copy_service.cc:140] P 08d7a0680d244b55b93d17f9ab57b84c: Received BeginTabletCopySession request for tablet 53b7eec870c44999b48f0bf744b2ffcd from peer d1406e92aec84ac585304e57bd37a40d ({username='slave'} at 127.24.226.136:57795)
I20240502 21:26:51.557195 31817 tablet_copy_service.cc:161] P 08d7a0680d244b55b93d17f9ab57b84c: Beginning new tablet copy session on tablet 53b7eec870c44999b48f0bf744b2ffcd from peer d1406e92aec84ac585304e57bd37a40d at {username='slave'} at 127.24.226.136:57795: session id = d1406e92aec84ac585304e57bd37a40d-53b7eec870c44999b48f0bf744b2ffcd
I20240502 21:26:51.561391 31817 tablet_copy_source_session.cc:205] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:51.562989 3656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b7eec870c44999b48f0bf744b2ffcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:51.572850 3644 tablet_copy_client.cc:308] T 62f083e9641042c990efcfb1c4104de5 P 9dec645b473f40578e726f3e50e219bf: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.132:35933
I20240502 21:26:51.573784 31947 tablet_copy_service.cc:140] P 2d50f33557944bb480ea32199d9ee4f1: Received BeginTabletCopySession request for tablet 62f083e9641042c990efcfb1c4104de5 from peer 9dec645b473f40578e726f3e50e219bf ({username='slave'} at 127.24.226.134:56721)
I20240502 21:26:51.574051 31947 tablet_copy_service.cc:161] P 2d50f33557944bb480ea32199d9ee4f1: Beginning new tablet copy session on tablet 62f083e9641042c990efcfb1c4104de5 from peer 9dec645b473f40578e726f3e50e219bf at {username='slave'} at 127.24.226.134:56721: session id = 9dec645b473f40578e726f3e50e219bf-62f083e9641042c990efcfb1c4104de5
I20240502 21:26:51.576066 3656 tablet_copy_client.cc:783] T 53b7eec870c44999b48f0bf744b2ffcd P d1406e92aec84ac585304e57bd37a40d: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:51.577025 3656 tablet_copy_client.cc:647] T 53b7eec870c44999b48f0bf744b2ffcd P d1406e92aec84ac585304e57bd37a40d: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:51.578665 3671 ts_tablet_manager.cc:902] T 5a54d2aad63c4f3182992f00b8564acf P d1406e92aec84ac585304e57bd37a40d: Initiating tablet copy from peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:51.578944 31947 tablet_copy_source_session.cc:205] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:51.579253 3656 tablet_copy_client.cc:523] T 53b7eec870c44999b48f0bf744b2ffcd P d1406e92aec84ac585304e57bd37a40d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:51.588057 3643 tablet_copy_client.cc:783] T 759b7529d7d84baab1a4622d34f803bf P 9dec645b473f40578e726f3e50e219bf: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:51.587901 3529 raft_consensus.cc:2932] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:51.593977 3656 tablet_bootstrap.cc:492] T 53b7eec870c44999b48f0bf744b2ffcd P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:51.594260 3681 tablet_bootstrap.cc:492] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:51.597069 3674 tablet_bootstrap.cc:492] T 2e4ed0121a404869b6fa33b14d3c49f8 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:51.599220 3674 tablet_bootstrap.cc:654] T 2e4ed0121a404869b6fa33b14d3c49f8 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:51.600149 3667 tablet_bootstrap.cc:492] T 2c737c49a32b4f3ea5ef735fa254d48e P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:51.601934 3667 tablet_bootstrap.cc:654] T 2c737c49a32b4f3ea5ef735fa254d48e P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:51.601584 31395 catalog_manager.cc:5484] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db reported cstate change: config changed from index 4 to 5, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:51.603387 3668 tablet_replica.cc:330] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 1e04157b98dd4fad98858c290b73c3db: stopping tablet replica
I20240502 21:26:51.603667 3668 raft_consensus.cc:2229] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:51.604003 3668 raft_consensus.cc:2258] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:51.604029 31797 raft_consensus.cc:2932] T b9342dbe0497499ba4c238f2ed033293 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:51.605020 3675 tablet_bootstrap.cc:492] T 2c737c49a32b4f3ea5ef735fa254d48e P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:51.605080 3668 ts_tablet_manager.cc:1875] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 1e04157b98dd4fad98858c290b73c3db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:51.606875 3675 tablet_bootstrap.cc:654] T 2c737c49a32b4f3ea5ef735fa254d48e P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:51.614435 31395 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:26:51.616863 3665 tablet_replica.cc:330] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:51.617125 3665 raft_consensus.cc:2229] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:51.617429 3665 raft_consensus.cc:2258] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:51.618386 3665 ts_tablet_manager.cc:1875] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:51.624282 3667 tablet_bootstrap.cc:492] T 2c737c49a32b4f3ea5ef735fa254d48e P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:51.624542 3667 ts_tablet_manager.cc:1367] T 2c737c49a32b4f3ea5ef735fa254d48e P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20240502 21:26:51.625339 3667 raft_consensus.cc:348] T 2c737c49a32b4f3ea5ef735fa254d48e P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:51.625941 3667 raft_consensus.cc:374] T 2c737c49a32b4f3ea5ef735fa254d48e P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:51.626170 3667 raft_consensus.cc:729] T 2c737c49a32b4f3ea5ef735fa254d48e P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:51.626495 3667 consensus_queue.cc:260] T 2c737c49a32b4f3ea5ef735fa254d48e P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:51.627321 3667 ts_tablet_manager.cc:1398] T 2c737c49a32b4f3ea5ef735fa254d48e P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:51.627637 3681 tablet_bootstrap.cc:654] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:51.631356 31665 raft_consensus.cc:2932] T b9342dbe0497499ba4c238f2ed033293 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:51.631753 31645 tablet_service.cc:1467] Processing CreateTablet for tablet 71974e7442b042bebc999b42833be63f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9074dd1629f54b0e9607336d3f7224a2]), partition=
I20240502 21:26:51.631938 3616 ts_tablet_manager.cc:902] T 88189733d9a14560971dbeb71543073f P 1ade0354c44c4c37ae480c7a94b11e1b: Initiating tablet copy from peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:51.632099 31645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71974e7442b042bebc999b42833be63f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:51.637837 3664 tablet_bootstrap.cc:492] T 2e4ed0121a404869b6fa33b14d3c49f8 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:51.640018 3665 ts_tablet_manager.cc:1888] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:51.640296 3664 ts_tablet_manager.cc:1367] T 2e4ed0121a404869b6fa33b14d3c49f8 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.138s user 0.000s sys 0.007s
I20240502 21:26:51.640867 3667 tablet_bootstrap.cc:492] T 3e6657723fe84e749dea768de44638e7 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:51.641614 3664 raft_consensus.cc:348] T 2e4ed0121a404869b6fa33b14d3c49f8 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:51.642414 3665 log.cc:1196] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/1ca6669bb60a4b7eb7b01acb8fc64800
I20240502 21:26:51.643759 3665 ts_tablet_manager.cc:1909] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:26:51.646768 3672 tablet_bootstrap.cc:492] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:51.646776 3664 raft_consensus.cc:374] T 2e4ed0121a404869b6fa33b14d3c49f8 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:51.646895 3664 raft_consensus.cc:729] T 2e4ed0121a404869b6fa33b14d3c49f8 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:51.647419 3675 tablet_bootstrap.cc:492] T 2c737c49a32b4f3ea5ef735fa254d48e P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:51.647956 3672 ts_tablet_manager.cc:1367] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.106s user 0.005s sys 0.014s
I20240502 21:26:51.648000 3664 consensus_queue.cc:260] T 2e4ed0121a404869b6fa33b14d3c49f8 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:51.648316 3675 ts_tablet_manager.cc:1367] T 2c737c49a32b4f3ea5ef735fa254d48e P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.000s
I20240502 21:26:51.649937 3675 raft_consensus.cc:348] T 2c737c49a32b4f3ea5ef735fa254d48e P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:51.650593 3667 tablet_bootstrap.cc:654] T 3e6657723fe84e749dea768de44638e7 P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:51.652487 3675 raft_consensus.cc:374] T 2c737c49a32b4f3ea5ef735fa254d48e P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:51.652791 3675 raft_consensus.cc:729] T 2c737c49a32b4f3ea5ef735fa254d48e P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:51.652963 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet 1ca6669bb60a4b7eb7b01acb8fc64800 (table rebalancer_dynamic_table_new_name [id=2bf308935f084e238190eddf14154196]) successfully deleted
I20240502 21:26:51.653314 3675 consensus_queue.cc:260] T 2c737c49a32b4f3ea5ef735fa254d48e P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:51.654193 3664 ts_tablet_manager.cc:1398] T 2e4ed0121a404869b6fa33b14d3c49f8 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20240502 21:26:51.654675 3689 tablet_bootstrap.cc:492] T 71974e7442b042bebc999b42833be63f P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:51.655092 3672 raft_consensus.cc:348] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:51.656623 3689 tablet_bootstrap.cc:654] T 71974e7442b042bebc999b42833be63f P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:51.657230 3672 raft_consensus.cc:374] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:51.657769 3681 tablet_bootstrap.cc:492] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:51.657665 3672 raft_consensus.cc:729] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:51.657966 3681 ts_tablet_manager.cc:1367] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.064s user 0.002s sys 0.003s
I20240502 21:26:51.658393 3672 consensus_queue.cc:260] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:51.658741 3681 raft_consensus.cc:348] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:51.658931 3681 raft_consensus.cc:374] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:51.659009 3681 raft_consensus.cc:729] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:51.659231 3681 consensus_queue.cc:260] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:51.660739 3681 ts_tablet_manager.cc:1398] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 21:26:51.589845 3644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62f083e9641042c990efcfb1c4104de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:51.632869 1945 tablet_service.cc:1467] Processing CreateTablet for tablet 71974e7442b042bebc999b42833be63f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9074dd1629f54b0e9607336d3f7224a2]), partition=
I20240502 21:26:51.664821 1945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71974e7442b042bebc999b42833be63f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:51.666296 3672 ts_tablet_manager.cc:1398] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.018s user 0.000s sys 0.002s
I20240502 21:26:51.668783 3672 tablet_bootstrap.cc:492] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:51.672726 3675 ts_tablet_manager.cc:1398] T 2c737c49a32b4f3ea5ef735fa254d48e P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20240502 21:26:51.677085 3644 tablet_copy_client.cc:783] T 62f083e9641042c990efcfb1c4104de5 P 9dec645b473f40578e726f3e50e219bf: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:51.677348 3644 tablet_copy_client.cc:647] T 62f083e9641042c990efcfb1c4104de5 P 9dec645b473f40578e726f3e50e219bf: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:51.679996 3672 tablet_bootstrap.cc:654] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:51.680131 3643 tablet_copy_client.cc:647] T 759b7529d7d84baab1a4622d34f803bf P 9dec645b473f40578e726f3e50e219bf: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:51.680325 3697 tablet_bootstrap.cc:492] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:51.682556 3643 tablet_copy_client.cc:523] T 759b7529d7d84baab1a4622d34f803bf P 9dec645b473f40578e726f3e50e219bf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:51.683326 3644 tablet_copy_client.cc:523] T 62f083e9641042c990efcfb1c4104de5 P 9dec645b473f40578e726f3e50e219bf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:51.687544 3643 tablet_bootstrap.cc:492] T 759b7529d7d84baab1a4622d34f803bf P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:51.688743 3697 tablet_bootstrap.cc:654] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:51.690325 3644 tablet_bootstrap.cc:492] T 62f083e9641042c990efcfb1c4104de5 P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:51.691486 3667 tablet_bootstrap.cc:492] T 3e6657723fe84e749dea768de44638e7 P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:51.691962 3667 ts_tablet_manager.cc:1367] T 3e6657723fe84e749dea768de44638e7 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.000s
I20240502 21:26:51.693907 3651 tablet_copy_client.cc:308] T a1fd14de204c40d98027ecb329ff79f3 P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.131:40807
I20240502 21:26:51.694816 31817 tablet_copy_service.cc:140] P 08d7a0680d244b55b93d17f9ab57b84c: Received BeginTabletCopySession request for tablet a1fd14de204c40d98027ecb329ff79f3 from peer 1e04157b98dd4fad98858c290b73c3db ({username='slave'} at 127.24.226.133:41593)
I20240502 21:26:51.695046 31817 tablet_copy_service.cc:161] P 08d7a0680d244b55b93d17f9ab57b84c: Beginning new tablet copy session on tablet a1fd14de204c40d98027ecb329ff79f3 from peer 1e04157b98dd4fad98858c290b73c3db at {username='slave'} at 127.24.226.133:41593: session id = 1e04157b98dd4fad98858c290b73c3db-a1fd14de204c40d98027ecb329ff79f3
I20240502 21:26:51.697885 3667 raft_consensus.cc:348] T 3e6657723fe84e749dea768de44638e7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:51.698742 31817 tablet_copy_source_session.cc:205] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:51.700138 3651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1fd14de204c40d98027ecb329ff79f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:51.701159 2210 tablet_service.cc:1467] Processing CreateTablet for tablet 71974e7442b042bebc999b42833be63f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9074dd1629f54b0e9607336d3f7224a2]), partition=
I20240502 21:26:51.701524 2210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71974e7442b042bebc999b42833be63f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:51.702826 3689 tablet_bootstrap.cc:492] T 71974e7442b042bebc999b42833be63f P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:51.702960 3689 ts_tablet_manager.cc:1367] T 71974e7442b042bebc999b42833be63f P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.048s user 0.007s sys 0.008s
I20240502 21:26:51.703619 3689 raft_consensus.cc:348] T 71974e7442b042bebc999b42833be63f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:51.703876 3689 raft_consensus.cc:374] T 71974e7442b042bebc999b42833be63f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:51.703960 3689 raft_consensus.cc:729] T 71974e7442b042bebc999b42833be63f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:51.704200 3689 consensus_queue.cc:260] T 71974e7442b042bebc999b42833be63f P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:51.707640 3689 ts_tablet_manager.cc:1398] T 71974e7442b042bebc999b42833be63f P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 21:26:51.709216 3666 tablet_replica.cc:330] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d: stopping tablet replica
I20240502 21:26:51.709424 3666 raft_consensus.cc:2229] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:51.710356 3697 tablet_bootstrap.cc:492] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:26:51.710567 3697 ts_tablet_manager.cc:1367] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.000s
I20240502 21:26:51.711457 3666 raft_consensus.cc:2258] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:51.711606 3697 raft_consensus.cc:348] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:51.711861 3697 raft_consensus.cc:374] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:51.711969 3697 raft_consensus.cc:729] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:26:51.712175 3697 consensus_queue.cc:260] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:51.713676 3697 ts_tablet_manager.cc:1398] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:51.714310 3666 ts_tablet_manager.cc:1875] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:51.714399 3672 tablet_bootstrap.cc:492] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:51.714572 3672 ts_tablet_manager.cc:1367] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.046s user 0.000s sys 0.006s
I20240502 21:26:51.715425 3672 raft_consensus.cc:348] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:51.715618 3672 raft_consensus.cc:374] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:51.715688 3672 raft_consensus.cc:729] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:51.715950 3672 consensus_queue.cc:260] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:51.719313 3672 ts_tablet_manager.cc:1398] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 21:26:51.727880 3666 ts_tablet_manager.cc:1888] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:51.728021 3651 tablet_copy_client.cc:783] T a1fd14de204c40d98027ecb329ff79f3 P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:51.728060 3666 log.cc:1196] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal/wals/1ca6669bb60a4b7eb7b01acb8fc64800
I20240502 21:26:51.729390 3651 tablet_copy_client.cc:647] T a1fd14de204c40d98027ecb329ff79f3 P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:51.729529 3666 ts_tablet_manager.cc:1909] T 1ca6669bb60a4b7eb7b01acb8fc64800 P d1406e92aec84ac585304e57bd37a40d: Deleting consensus metadata
I20240502 21:26:51.730701 31381 catalog_manager.cc:4830] TS d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): tablet 1ca6669bb60a4b7eb7b01acb8fc64800 (table rebalancer_dynamic_table_new_name [id=2bf308935f084e238190eddf14154196]) successfully deleted
I20240502 21:26:51.731041 3651 tablet_copy_client.cc:523] T a1fd14de204c40d98027ecb329ff79f3 P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:51.733846 3687 raft_consensus.cc:482] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:51.733987 3687 raft_consensus.cc:504] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:51.734817 3687 leader_election.cc:288] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:51.735344 31795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c737c49a32b4f3ea5ef735fa254d48e" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:51.735576 31795 raft_consensus.cc:2454] T 2c737c49a32b4f3ea5ef735fa254d48e P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 0.
I20240502 21:26:51.736392 1769 leader_election.cc:302] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 1e04157b98dd4fad98858c290b73c3db; no voters:
I20240502 21:26:51.736850 3651 tablet_bootstrap.cc:492] T a1fd14de204c40d98027ecb329ff79f3 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:51.736914 2095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c737c49a32b4f3ea5ef735fa254d48e" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:51.737155 2095 raft_consensus.cc:2454] T 2c737c49a32b4f3ea5ef735fa254d48e P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 0.
I20240502 21:26:51.738772 3674 tablet_bootstrap.cc:492] T 2e4ed0121a404869b6fa33b14d3c49f8 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:51.738976 3674 ts_tablet_manager.cc:1367] T 2e4ed0121a404869b6fa33b14d3c49f8 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.142s user 0.003s sys 0.004s
I20240502 21:26:51.739840 3674 raft_consensus.cc:348] T 2e4ed0121a404869b6fa33b14d3c49f8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:51.740090 3674 raft_consensus.cc:374] T 2e4ed0121a404869b6fa33b14d3c49f8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:51.740239 3674 raft_consensus.cc:729] T 2e4ed0121a404869b6fa33b14d3c49f8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:51.740521 3674 consensus_queue.cc:260] T 2e4ed0121a404869b6fa33b14d3c49f8 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:51.741099 3674 ts_tablet_manager.cc:1398] T 2e4ed0121a404869b6fa33b14d3c49f8 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:51.743119 3687 raft_consensus.cc:2781] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:51.743245 3687 raft_consensus.cc:482] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:51.743320 3687 raft_consensus.cc:3037] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:51.746508 3687 raft_consensus.cc:504] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:51.749691 2095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c737c49a32b4f3ea5ef735fa254d48e" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:26:51.749965 2095 raft_consensus.cc:3037] T 2c737c49a32b4f3ea5ef735fa254d48e P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:51.756242 2096 raft_consensus.cc:1203] T dab4f06cc78946dc962267cb2aae04f7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:26:51.698087 3667 raft_consensus.cc:374] T 3e6657723fe84e749dea768de44638e7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:51.756736 3667 raft_consensus.cc:729] T 3e6657723fe84e749dea768de44638e7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:51.756987 3667 consensus_queue.cc:260] T 3e6657723fe84e749dea768de44638e7 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:51.757539 3667 ts_tablet_manager.cc:1398] T 3e6657723fe84e749dea768de44638e7 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.060s user 0.001s sys 0.000s
I20240502 21:26:51.758738 3672 tablet_bootstrap.cc:492] T 2e4ed0121a404869b6fa33b14d3c49f8 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:51.763108 3668 ts_tablet_manager.cc:1888] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 1e04157b98dd4fad98858c290b73c3db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:51.763298 3668 log.cc:1196] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 1e04157b98dd4fad98858c290b73c3db: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal/wals/1ca6669bb60a4b7eb7b01acb8fc64800
I20240502 21:26:51.766614 3695 ts_tablet_manager.cc:902] T c5c344f8f84f4a73b853b4f908b93964 P 1ade0354c44c4c37ae480c7a94b11e1b: Initiating tablet copy from peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:51.768232 3702 ts_tablet_manager.cc:902] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 2d50f33557944bb480ea32199d9ee4f1: Initiating tablet copy from peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:51.769074 3616 tablet_copy_client.cc:308] T 88189733d9a14560971dbeb71543073f P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.132:35933
I20240502 21:26:51.769749 2095 raft_consensus.cc:2454] T 2c737c49a32b4f3ea5ef735fa254d48e P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:26:51.769804 31947 tablet_copy_service.cc:140] P 2d50f33557944bb480ea32199d9ee4f1: Received BeginTabletCopySession request for tablet 88189733d9a14560971dbeb71543073f from peer 1ade0354c44c4c37ae480c7a94b11e1b ({username='slave'} at 127.24.226.135:33163)
I20240502 21:26:51.769975 31947 tablet_copy_service.cc:161] P 2d50f33557944bb480ea32199d9ee4f1: Beginning new tablet copy session on tablet 88189733d9a14560971dbeb71543073f from peer 1ade0354c44c4c37ae480c7a94b11e1b at {username='slave'} at 127.24.226.135:33163: session id = 1ade0354c44c4c37ae480c7a94b11e1b-88189733d9a14560971dbeb71543073f
I20240502 21:26:51.770767 3672 tablet_bootstrap.cc:654] T 2e4ed0121a404869b6fa33b14d3c49f8 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:51.771826 3668 ts_tablet_manager.cc:1909] T 1ca6669bb60a4b7eb7b01acb8fc64800 P 1e04157b98dd4fad98858c290b73c3db: Deleting consensus metadata
I20240502 21:26:51.772105 31535 consensus_queue.cc:237] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } }
I20240502 21:26:51.773015 31947 tablet_copy_source_session.cc:205] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:51.773633 31381 catalog_manager.cc:4830] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): tablet 1ca6669bb60a4b7eb7b01acb8fc64800 (table rebalancer_dynamic_table_new_name [id=2bf308935f084e238190eddf14154196]) successfully deleted
I20240502 21:26:51.781214 31795 raft_consensus.cc:1261] T 8ab82de23912483ea178376a5ba5cdc1 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:51.781839 3207 consensus_queue.cc:1035] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:51.783918 31666 raft_consensus.cc:1261] T 8ab82de23912483ea178376a5ba5cdc1 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:51.785058 3553 consensus_queue.cc:1035] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:51.790171 3644 tablet_bootstrap.cc:492] T 62f083e9641042c990efcfb1c4104de5 P 9dec645b473f40578e726f3e50e219bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:51.792980 3553 raft_consensus.cc:2932] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } } }
I20240502 21:26:51.796528 3672 tablet_bootstrap.cc:492] T 2e4ed0121a404869b6fa33b14d3c49f8 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:51.796744 3672 ts_tablet_manager.cc:1367] T 2e4ed0121a404869b6fa33b14d3c49f8 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.038s user 0.000s sys 0.004s
I20240502 21:26:51.796743 31795 raft_consensus.cc:2932] T 8ab82de23912483ea178376a5ba5cdc1 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } } }
I20240502 21:26:51.797874 3672 raft_consensus.cc:348] T 2e4ed0121a404869b6fa33b14d3c49f8 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:51.798170 3672 raft_consensus.cc:374] T 2e4ed0121a404869b6fa33b14d3c49f8 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:51.798357 3672 raft_consensus.cc:729] T 2e4ed0121a404869b6fa33b14d3c49f8 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:51.798652 3672 consensus_queue.cc:260] T 2e4ed0121a404869b6fa33b14d3c49f8 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:51.800146 3672 ts_tablet_manager.cc:1398] T 2e4ed0121a404869b6fa33b14d3c49f8 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:26:51.802165 3644 tablet_bootstrap.cc:492] T 62f083e9641042c990efcfb1c4104de5 P 9dec645b473f40578e726f3e50e219bf: Bootstrap complete.
I20240502 21:26:51.802393 3644 ts_tablet_manager.cc:1367] T 62f083e9641042c990efcfb1c4104de5 P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.113s user 0.047s sys 0.000s
I20240502 21:26:51.802382 31390 catalog_manager.cc:5484] T 8ab82de23912483ea178376a5ba5cdc1 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index -1 to 4, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } } }
I20240502 21:26:51.803089 3644 raft_consensus.cc:348] T 62f083e9641042c990efcfb1c4104de5 P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:26:51.803359 3644 raft_consensus.cc:729] T 62f083e9641042c990efcfb1c4104de5 P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: LEARNER
I20240502 21:26:51.803587 3644 consensus_queue.cc:260] T 62f083e9641042c990efcfb1c4104de5 P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:26:51.803941 2816 raft_consensus.cc:482] T 2e4ed0121a404869b6fa33b14d3c49f8 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:51.804109 2816 raft_consensus.cc:504] T 2e4ed0121a404869b6fa33b14d3c49f8 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:51.804795 2816 leader_election.cc:288] T 2e4ed0121a404869b6fa33b14d3c49f8 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:51.805632 3644 ts_tablet_manager.cc:1398] T 62f083e9641042c990efcfb1c4104de5 P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:51.806180 31947 tablet_copy_service.cc:342] P 2d50f33557944bb480ea32199d9ee4f1: Request end of tablet copy session 9dec645b473f40578e726f3e50e219bf-62f083e9641042c990efcfb1c4104de5 received from {username='slave'} at 127.24.226.134:56721
I20240502 21:26:51.806329 31947 tablet_copy_service.cc:433] P 2d50f33557944bb480ea32199d9ee4f1: ending tablet copy session 9dec645b473f40578e726f3e50e219bf-62f083e9641042c990efcfb1c4104de5 on tablet 62f083e9641042c990efcfb1c4104de5 with peer 9dec645b473f40578e726f3e50e219bf
I20240502 21:26:51.806205 31924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4ed0121a404869b6fa33b14d3c49f8" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:51.807164 31924 raft_consensus.cc:2454] T 2e4ed0121a404869b6fa33b14d3c49f8 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 0.
I20240502 21:26:51.808043 1770 leader_election.cc:302] T 2e4ed0121a404869b6fa33b14d3c49f8 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:51.809531 2816 raft_consensus.cc:2781] T 2e4ed0121a404869b6fa33b14d3c49f8 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:51.809628 2816 raft_consensus.cc:482] T 2e4ed0121a404869b6fa33b14d3c49f8 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:51.809731 2816 raft_consensus.cc:3037] T 2e4ed0121a404869b6fa33b14d3c49f8 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:51.812278 2816 raft_consensus.cc:504] T 2e4ed0121a404869b6fa33b14d3c49f8 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:51.814103 31924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4ed0121a404869b6fa33b14d3c49f8" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:51.814313 31924 raft_consensus.cc:3037] T 2e4ed0121a404869b6fa33b14d3c49f8 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:51.818032 31924 raft_consensus.cc:2454] T 2e4ed0121a404869b6fa33b14d3c49f8 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:26:51.818560 1770 leader_election.cc:302] T 2e4ed0121a404869b6fa33b14d3c49f8 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:51.818928 3529 raft_consensus.cc:2781] T 2e4ed0121a404869b6fa33b14d3c49f8 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:51.819105 3529 raft_consensus.cc:686] T 2e4ed0121a404869b6fa33b14d3c49f8 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Becoming Leader. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Running, Role: LEADER
I20240502 21:26:51.819412 3529 consensus_queue.cc:237] T 2e4ed0121a404869b6fa33b14d3c49f8 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:51.821060 2816 leader_election.cc:288] T 2e4ed0121a404869b6fa33b14d3c49f8 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Requested vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:51.833117 1769 leader_election.cc:302] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 1e04157b98dd4fad98858c290b73c3db; no voters:
I20240502 21:26:51.833760 2816 raft_consensus.cc:2781] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:51.834026 2816 raft_consensus.cc:686] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Becoming Leader. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Running, Role: LEADER
I20240502 21:26:51.834420 2816 consensus_queue.cc:237] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:51.833801 3644 ts_tablet_manager.cc:902] T ae56d18c629a4e5eb50911c89de4227c P 9dec645b473f40578e726f3e50e219bf: Initiating tablet copy from peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:51.838203 31666 raft_consensus.cc:2932] T 8ab82de23912483ea178376a5ba5cdc1 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } } }
I20240502 21:26:51.774927 3616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88189733d9a14560971dbeb71543073f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:51.845945 3687 leader_election.cc:288] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Requested vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:51.846206 31795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c737c49a32b4f3ea5ef735fa254d48e" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:51.846421 31795 raft_consensus.cc:3037] T 2c737c49a32b4f3ea5ef735fa254d48e P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:51.849857 31795 raft_consensus.cc:2454] T 2c737c49a32b4f3ea5ef735fa254d48e P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:26:51.853658 3728 ts_tablet_manager.cc:902] T acb3bdc518b84b5abc362cc4e12027c3 P a44f41b0b6644fb6aa29476aed8c964f: Initiating tablet copy from peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:26:51.861459 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4ed0121a404869b6fa33b14d3c49f8" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:26:51.861759 2227 raft_consensus.cc:3037] T 2e4ed0121a404869b6fa33b14d3c49f8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:51.862730 3674 tablet_bootstrap.cc:492] T 71974e7442b042bebc999b42833be63f P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:51.862813 2231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4ed0121a404869b6fa33b14d3c49f8" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
W20240502 21:26:51.863621 31471 consensus_peers.cc:485] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f -> Peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): Couldn't send request to peer d1406e92aec84ac585304e57bd37a40d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ab82de23912483ea178376a5ba5cdc1. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:51.865604 2227 raft_consensus.cc:2454] T 2e4ed0121a404869b6fa33b14d3c49f8 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:26:51.867059 3726 ts_tablet_manager.cc:902] T 5b1dcae186f542cea92c9191fa8ef951 P 1e04157b98dd4fad98858c290b73c3db: Initiating tablet copy from peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:51.869375 3702 tablet_copy_client.cc:308] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 2d50f33557944bb480ea32199d9ee4f1: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.129:41039
I20240502 21:26:51.870047 31556 tablet_copy_service.cc:140] P a44f41b0b6644fb6aa29476aed8c964f: Received BeginTabletCopySession request for tablet 5bc6c7211d5f4f73b2a050b9ec78c6b1 from peer 2d50f33557944bb480ea32199d9ee4f1 ({username='slave'} at 127.24.226.132:42427)
I20240502 21:26:51.870215 31556 tablet_copy_service.cc:161] P a44f41b0b6644fb6aa29476aed8c964f: Beginning new tablet copy session on tablet 5bc6c7211d5f4f73b2a050b9ec78c6b1 from peer 2d50f33557944bb480ea32199d9ee4f1 at {username='slave'} at 127.24.226.132:42427: session id = 2d50f33557944bb480ea32199d9ee4f1-5bc6c7211d5f4f73b2a050b9ec78c6b1
I20240502 21:26:51.871675 3616 tablet_copy_client.cc:783] T 88189733d9a14560971dbeb71543073f P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:51.871904 3616 tablet_copy_client.cc:647] T 88189733d9a14560971dbeb71543073f P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:51.872738 3674 tablet_bootstrap.cc:654] T 71974e7442b042bebc999b42833be63f P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:51.873668 31395 catalog_manager.cc:5484] T 2e4ed0121a404869b6fa33b14d3c49f8 P 1e04157b98dd4fad98858c290b73c3db reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e04157b98dd4fad98858c290b73c3db (127.24.226.133). New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:51.875777 31556 tablet_copy_source_session.cc:205] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:51.877707 3702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc6c7211d5f4f73b2a050b9ec78c6b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:51.881179 31395 catalog_manager.cc:5484] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e04157b98dd4fad98858c290b73c3db (127.24.226.133). New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:51.891247 1966 raft_consensus.cc:1203] T 62f083e9641042c990efcfb1c4104de5 P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:26:51.896456 3616 tablet_copy_client.cc:523] T 88189733d9a14560971dbeb71543073f P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:51.899138 3708 ts_tablet_manager.cc:902] T 55fc18e4cc98446ca50c1dd2b6a9a89d P d1406e92aec84ac585304e57bd37a40d: Initiating tablet copy from peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:51.900090 3674 tablet_bootstrap.cc:492] T 71974e7442b042bebc999b42833be63f P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:51.900544 3674 ts_tablet_manager.cc:1367] T 71974e7442b042bebc999b42833be63f P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.001s
I20240502 21:26:51.902300 3695 tablet_copy_client.cc:308] T c5c344f8f84f4a73b853b4f908b93964 P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.131:40807
I20240502 21:26:51.902798 3674 raft_consensus.cc:348] T 71974e7442b042bebc999b42833be63f P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:51.911679 3616 tablet_bootstrap.cc:492] T 88189733d9a14560971dbeb71543073f P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:51.912307 3674 raft_consensus.cc:374] T 71974e7442b042bebc999b42833be63f P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:51.914860 3671 tablet_copy_client.cc:308] T 5a54d2aad63c4f3182992f00b8564acf P d1406e92aec84ac585304e57bd37a40d: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.131:40807
I20240502 21:26:51.916903 31817 tablet_copy_service.cc:140] P 08d7a0680d244b55b93d17f9ab57b84c: Received BeginTabletCopySession request for tablet 5a54d2aad63c4f3182992f00b8564acf from peer d1406e92aec84ac585304e57bd37a40d ({username='slave'} at 127.24.226.136:57795)
I20240502 21:26:51.917075 31817 tablet_copy_service.cc:161] P 08d7a0680d244b55b93d17f9ab57b84c: Beginning new tablet copy session on tablet 5a54d2aad63c4f3182992f00b8564acf from peer d1406e92aec84ac585304e57bd37a40d at {username='slave'} at 127.24.226.136:57795: session id = d1406e92aec84ac585304e57bd37a40d-5a54d2aad63c4f3182992f00b8564acf
I20240502 21:26:51.919826 31817 tablet_copy_source_session.cc:205] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:51.920817 3728 tablet_copy_client.cc:308] T acb3bdc518b84b5abc362cc4e12027c3 P a44f41b0b6644fb6aa29476aed8c964f: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.134:43753
I20240502 21:26:51.912408 3674 raft_consensus.cc:729] T 71974e7442b042bebc999b42833be63f P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:51.921486 1986 tablet_copy_service.cc:140] P 9dec645b473f40578e726f3e50e219bf: Received BeginTabletCopySession request for tablet acb3bdc518b84b5abc362cc4e12027c3 from peer a44f41b0b6644fb6aa29476aed8c964f ({username='slave'} at 127.24.226.129:41327)
I20240502 21:26:51.921663 1986 tablet_copy_service.cc:161] P 9dec645b473f40578e726f3e50e219bf: Beginning new tablet copy session on tablet acb3bdc518b84b5abc362cc4e12027c3 from peer a44f41b0b6644fb6aa29476aed8c964f at {username='slave'} at 127.24.226.129:41327: session id = a44f41b0b6644fb6aa29476aed8c964f-acb3bdc518b84b5abc362cc4e12027c3
I20240502 21:26:51.921753 3644 tablet_copy_client.cc:308] T ae56d18c629a4e5eb50911c89de4227c P 9dec645b473f40578e726f3e50e219bf: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.132:35933
I20240502 21:26:51.921708 3674 consensus_queue.cc:260] T 71974e7442b042bebc999b42833be63f P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:51.922233 3674 ts_tablet_manager.cc:1398] T 71974e7442b042bebc999b42833be63f P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.021s user 0.001s sys 0.000s
I20240502 21:26:51.922279 31947 tablet_copy_service.cc:140] P 2d50f33557944bb480ea32199d9ee4f1: Received BeginTabletCopySession request for tablet ae56d18c629a4e5eb50911c89de4227c from peer 9dec645b473f40578e726f3e50e219bf ({username='slave'} at 127.24.226.134:56721)
I20240502 21:26:51.922448 31947 tablet_copy_service.cc:161] P 2d50f33557944bb480ea32199d9ee4f1: Beginning new tablet copy session on tablet ae56d18c629a4e5eb50911c89de4227c from peer 9dec645b473f40578e726f3e50e219bf at {username='slave'} at 127.24.226.134:56721: session id = 9dec645b473f40578e726f3e50e219bf-ae56d18c629a4e5eb50911c89de4227c
I20240502 21:26:51.923758 2816 raft_consensus.cc:482] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:51.923902 2816 raft_consensus.cc:504] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:51.924938 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e6657723fe84e749dea768de44638e7" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:51.925150 31535 raft_consensus.cc:2454] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 0.
I20240502 21:26:51.925283 31947 tablet_copy_source_session.cc:205] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:51.924424 1986 tablet_copy_source_session.cc:205] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:51.925612 1770 leader_election.cc:302] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:51.925894 3529 raft_consensus.cc:2781] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:51.925985 3529 raft_consensus.cc:482] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:51.926090 3529 raft_consensus.cc:3037] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:51.926370 3671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a54d2aad63c4f3182992f00b8564acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:51.926715 3728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acb3bdc518b84b5abc362cc4e12027c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:51.928807 3529 raft_consensus.cc:504] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:51.929602 3702 tablet_copy_client.cc:783] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 2d50f33557944bb480ea32199d9ee4f1: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:51.929965 2097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e6657723fe84e749dea768de44638e7" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:26:51.930210 2097 raft_consensus.cc:3037] T 3e6657723fe84e749dea768de44638e7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:51.929915 3702 tablet_copy_client.cc:647] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 2d50f33557944bb480ea32199d9ee4f1: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:51.932571 3529 leader_election.cc:288] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Requested vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:51.932782 3644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae56d18c629a4e5eb50911c89de4227c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:51.933571 2097 raft_consensus.cc:2454] T 3e6657723fe84e749dea768de44638e7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:26:51.934060 1769 leader_election.cc:302] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 1e04157b98dd4fad98858c290b73c3db; no voters:
I20240502 21:26:51.934365 3702 tablet_copy_client.cc:523] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 2d50f33557944bb480ea32199d9ee4f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:51.934541 3529 raft_consensus.cc:2781] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:51.934751 3529 raft_consensus.cc:686] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Becoming Leader. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Running, Role: LEADER
I20240502 21:26:51.935129 3529 consensus_queue.cc:237] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:51.937073 2097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e6657723fe84e749dea768de44638e7" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:51.937268 2097 raft_consensus.cc:2362] T 3e6657723fe84e749dea768de44638e7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1. Re-sending same reply.
I20240502 21:26:51.937762 2816 leader_election.cc:288] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:51.938302 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e6657723fe84e749dea768de44638e7" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:26:51.938627 31535 raft_consensus.cc:3037] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:51.941409 31535 raft_consensus.cc:2454] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:26:51.946844 3671 tablet_copy_client.cc:783] T 5a54d2aad63c4f3182992f00b8564acf P d1406e92aec84ac585304e57bd37a40d: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:51.947151 3671 tablet_copy_client.cc:647] T 5a54d2aad63c4f3182992f00b8564acf P d1406e92aec84ac585304e57bd37a40d: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:51.947083 31389 catalog_manager.cc:5484] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e04157b98dd4fad98858c290b73c3db (127.24.226.133). New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:51.950474 3671 tablet_copy_client.cc:523] T 5a54d2aad63c4f3182992f00b8564acf P d1406e92aec84ac585304e57bd37a40d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:51.937826 3702 tablet_bootstrap.cc:492] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:51.950740 3708 tablet_copy_client.cc:308] T 55fc18e4cc98446ca50c1dd2b6a9a89d P d1406e92aec84ac585304e57bd37a40d: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.129:41039
I20240502 21:26:51.952247 31556 tablet_copy_service.cc:140] P a44f41b0b6644fb6aa29476aed8c964f: Received BeginTabletCopySession request for tablet 55fc18e4cc98446ca50c1dd2b6a9a89d from peer d1406e92aec84ac585304e57bd37a40d ({username='slave'} at 127.24.226.136:52831)
I20240502 21:26:51.952353 3726 tablet_copy_client.cc:308] T 5b1dcae186f542cea92c9191fa8ef951 P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.131:40807
I20240502 21:26:51.952426 31556 tablet_copy_service.cc:161] P a44f41b0b6644fb6aa29476aed8c964f: Beginning new tablet copy session on tablet 55fc18e4cc98446ca50c1dd2b6a9a89d from peer d1406e92aec84ac585304e57bd37a40d at {username='slave'} at 127.24.226.136:52831: session id = d1406e92aec84ac585304e57bd37a40d-55fc18e4cc98446ca50c1dd2b6a9a89d
I20240502 21:26:51.952914 3644 tablet_copy_client.cc:783] T ae56d18c629a4e5eb50911c89de4227c P 9dec645b473f40578e726f3e50e219bf: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:51.953127 3644 tablet_copy_client.cc:647] T ae56d18c629a4e5eb50911c89de4227c P 9dec645b473f40578e726f3e50e219bf: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:51.953249 3728 tablet_copy_client.cc:783] T acb3bdc518b84b5abc362cc4e12027c3 P a44f41b0b6644fb6aa29476aed8c964f: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:51.953486 3728 tablet_copy_client.cc:647] T acb3bdc518b84b5abc362cc4e12027c3 P a44f41b0b6644fb6aa29476aed8c964f: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:51.953483 31817 tablet_copy_service.cc:140] P 08d7a0680d244b55b93d17f9ab57b84c: Received BeginTabletCopySession request for tablet 5b1dcae186f542cea92c9191fa8ef951 from peer 1e04157b98dd4fad98858c290b73c3db ({username='slave'} at 127.24.226.133:41593)
I20240502 21:26:51.953622 31817 tablet_copy_service.cc:161] P 08d7a0680d244b55b93d17f9ab57b84c: Beginning new tablet copy session on tablet 5b1dcae186f542cea92c9191fa8ef951 from peer 1e04157b98dd4fad98858c290b73c3db at {username='slave'} at 127.24.226.133:41593: session id = 1e04157b98dd4fad98858c290b73c3db-5b1dcae186f542cea92c9191fa8ef951
I20240502 21:26:51.955117 3644 tablet_copy_client.cc:523] T ae56d18c629a4e5eb50911c89de4227c P 9dec645b473f40578e726f3e50e219bf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:51.955346 31556 tablet_copy_source_session.cc:205] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:51.957448 31817 tablet_copy_source_session.cc:205] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:51.958783 3708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55fc18e4cc98446ca50c1dd2b6a9a89d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:51.958806 3726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b1dcae186f542cea92c9191fa8ef951. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:51.960031 3728 tablet_copy_client.cc:523] T acb3bdc518b84b5abc362cc4e12027c3 P a44f41b0b6644fb6aa29476aed8c964f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:51.963398 3728 tablet_bootstrap.cc:492] T acb3bdc518b84b5abc362cc4e12027c3 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:51.965154 3610 raft_consensus.cc:482] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:51.965394 3610 raft_consensus.cc:504] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:51.965752 31817 tablet_copy_service.cc:140] P 08d7a0680d244b55b93d17f9ab57b84c: Received BeginTabletCopySession request for tablet c5c344f8f84f4a73b853b4f908b93964 from peer 1ade0354c44c4c37ae480c7a94b11e1b ({username='slave'} at 127.24.226.135:48019)
I20240502 21:26:51.965909 31817 tablet_copy_service.cc:161] P 08d7a0680d244b55b93d17f9ab57b84c: Beginning new tablet copy session on tablet c5c344f8f84f4a73b853b4f908b93964 from peer 1ade0354c44c4c37ae480c7a94b11e1b at {username='slave'} at 127.24.226.135:48019: session id = 1ade0354c44c4c37ae480c7a94b11e1b-c5c344f8f84f4a73b853b4f908b93964
I20240502 21:26:51.966115 3610 leader_election.cc:288] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:51.967608 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71974e7442b042bebc999b42833be63f" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:51.967830 31665 raft_consensus.cc:2454] T 71974e7442b042bebc999b42833be63f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 0.
I20240502 21:26:51.968521 31817 tablet_copy_source_session.cc:205] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:51.968515 1902 leader_election.cc:302] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:26:51.968997 3610 raft_consensus.cc:2781] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:51.969105 3610 raft_consensus.cc:482] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:51.969220 3610 raft_consensus.cc:3037] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:51.969761 3695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5c344f8f84f4a73b853b4f908b93964. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:51.969771 3671 tablet_bootstrap.cc:492] T 5a54d2aad63c4f3182992f00b8564acf P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:51.972256 2230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71974e7442b042bebc999b42833be63f" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:26:51.972501 2230 raft_consensus.cc:2454] T 71974e7442b042bebc999b42833be63f P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 0.
I20240502 21:26:51.973223 3610 raft_consensus.cc:504] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:51.974227 2230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71974e7442b042bebc999b42833be63f" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:26:51.974431 2230 raft_consensus.cc:3037] T 71974e7442b042bebc999b42833be63f P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:51.977759 3610 leader_election.cc:288] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 election: Requested vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:51.977813 3695 tablet_copy_client.cc:783] T c5c344f8f84f4a73b853b4f908b93964 P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:51.977847 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71974e7442b042bebc999b42833be63f" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:51.977977 2230 raft_consensus.cc:2454] T 71974e7442b042bebc999b42833be63f P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 1.
I20240502 21:26:51.978035 3695 tablet_copy_client.cc:647] T c5c344f8f84f4a73b853b4f908b93964 P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:51.978044 31665 raft_consensus.cc:3037] T 71974e7442b042bebc999b42833be63f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:51.979234 1901 leader_election.cc:302] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dec645b473f40578e726f3e50e219bf, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:51.979483 3708 tablet_copy_client.cc:783] T 55fc18e4cc98446ca50c1dd2b6a9a89d P d1406e92aec84ac585304e57bd37a40d: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:51.979602 3695 tablet_copy_client.cc:523] T c5c344f8f84f4a73b853b4f908b93964 P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:51.979549 3610 raft_consensus.cc:2781] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:51.979704 3708 tablet_copy_client.cc:647] T 55fc18e4cc98446ca50c1dd2b6a9a89d P d1406e92aec84ac585304e57bd37a40d: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:51.980085 3610 raft_consensus.cc:686] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf [term 1 LEADER]: Becoming Leader. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Running, Role: LEADER
I20240502 21:26:51.980334 3610 consensus_queue.cc:237] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:51.980971 31665 raft_consensus.cc:2454] T 71974e7442b042bebc999b42833be63f P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 1.
I20240502 21:26:51.981935 3708 tablet_copy_client.cc:523] T 55fc18e4cc98446ca50c1dd2b6a9a89d P d1406e92aec84ac585304e57bd37a40d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:51.982478 3695 tablet_bootstrap.cc:492] T c5c344f8f84f4a73b853b4f908b93964 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:51.983479 3644 tablet_bootstrap.cc:492] T ae56d18c629a4e5eb50911c89de4227c P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:51.995855 3723 ts_tablet_manager.cc:902] T b9342dbe0497499ba4c238f2ed033293 P 9dec645b473f40578e726f3e50e219bf: Initiating tablet copy from peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:26:52.008997 31389 catalog_manager.cc:5484] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dec645b473f40578e726f3e50e219bf (127.24.226.134). New cstate: current_term: 1 leader_uuid: "9dec645b473f40578e726f3e50e219bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:52.017413 3726 tablet_copy_client.cc:783] T 5b1dcae186f542cea92c9191fa8ef951 P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:52.018090 3726 tablet_copy_client.cc:647] T 5b1dcae186f542cea92c9191fa8ef951 P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:52.020579 3726 tablet_copy_client.cc:523] T 5b1dcae186f542cea92c9191fa8ef951 P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:52.026254 3723 tablet_copy_client.cc:308] T b9342dbe0497499ba4c238f2ed033293 P 9dec645b473f40578e726f3e50e219bf: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.133:39863
I20240502 21:26:52.026821 3708 tablet_bootstrap.cc:492] T 55fc18e4cc98446ca50c1dd2b6a9a89d P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:52.026968 1854 tablet_copy_service.cc:140] P 1e04157b98dd4fad98858c290b73c3db: Received BeginTabletCopySession request for tablet b9342dbe0497499ba4c238f2ed033293 from peer 9dec645b473f40578e726f3e50e219bf ({username='slave'} at 127.24.226.134:51927)
I20240502 21:26:52.027153 1854 tablet_copy_service.cc:161] P 1e04157b98dd4fad98858c290b73c3db: Beginning new tablet copy session on tablet b9342dbe0497499ba4c238f2ed033293 from peer 9dec645b473f40578e726f3e50e219bf at {username='slave'} at 127.24.226.134:51927: session id = 9dec645b473f40578e726f3e50e219bf-b9342dbe0497499ba4c238f2ed033293
I20240502 21:26:52.028309 3726 tablet_bootstrap.cc:492] T 5b1dcae186f542cea92c9191fa8ef951 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:52.028815 3728 tablet_bootstrap.cc:492] T acb3bdc518b84b5abc362cc4e12027c3 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:52.029284 3728 tablet_bootstrap.cc:492] T acb3bdc518b84b5abc362cc4e12027c3 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap complete.
I20240502 21:26:52.029526 3728 ts_tablet_manager.cc:1367] T acb3bdc518b84b5abc362cc4e12027c3 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.066s user 0.007s sys 0.004s
I20240502 21:26:52.030131 1854 tablet_copy_source_session.cc:205] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:52.035456 3723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9342dbe0497499ba4c238f2ed033293. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:52.038725 3643 tablet_bootstrap.cc:492] T 759b7529d7d84baab1a4622d34f803bf P 9dec645b473f40578e726f3e50e219bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:52.039121 3643 tablet_bootstrap.cc:492] T 759b7529d7d84baab1a4622d34f803bf P 9dec645b473f40578e726f3e50e219bf: Bootstrap complete.
I20240502 21:26:52.039381 3643 ts_tablet_manager.cc:1367] T 759b7529d7d84baab1a4622d34f803bf P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.352s user 0.040s sys 0.011s
I20240502 21:26:52.040107 3643 raft_consensus.cc:348] T 759b7529d7d84baab1a4622d34f803bf P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:26:52.040372 3643 raft_consensus.cc:729] T 759b7529d7d84baab1a4622d34f803bf P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: LEARNER
I20240502 21:26:52.046602 3702 tablet_bootstrap.cc:492] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:52.047549 3723 tablet_copy_client.cc:783] T b9342dbe0497499ba4c238f2ed033293 P 9dec645b473f40578e726f3e50e219bf: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:52.048038 3702 tablet_bootstrap.cc:492] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap complete.
I20240502 21:26:52.030336 3728 raft_consensus.cc:348] T acb3bdc518b84b5abc362cc4e12027c3 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } }
I20240502 21:26:52.048954 3728 raft_consensus.cc:729] T acb3bdc518b84b5abc362cc4e12027c3 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: LEARNER
I20240502 21:26:52.049240 3728 consensus_queue.cc:260] T acb3bdc518b84b5abc362cc4e12027c3 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } }
I20240502 21:26:52.049803 3702 ts_tablet_manager.cc:1367] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.112s user 0.008s sys 0.003s
I20240502 21:26:52.050067 3728 ts_tablet_manager.cc:1398] T acb3bdc518b84b5abc362cc4e12027c3 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.020s user 0.002s sys 0.001s
I20240502 21:26:52.053143 3702 raft_consensus.cc:348] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } }
I20240502 21:26:52.057824 3702 raft_consensus.cc:729] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: LEARNER
I20240502 21:26:52.058202 3702 consensus_queue.cc:260] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } }
I20240502 21:26:52.062566 3702 ts_tablet_manager.cc:1398] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20240502 21:26:52.064014 31556 tablet_copy_service.cc:342] P a44f41b0b6644fb6aa29476aed8c964f: Request end of tablet copy session 2d50f33557944bb480ea32199d9ee4f1-5bc6c7211d5f4f73b2a050b9ec78c6b1 received from {username='slave'} at 127.24.226.132:42427
I20240502 21:26:52.064143 31556 tablet_copy_service.cc:433] P a44f41b0b6644fb6aa29476aed8c964f: ending tablet copy session 2d50f33557944bb480ea32199d9ee4f1-5bc6c7211d5f4f73b2a050b9ec78c6b1 on tablet 5bc6c7211d5f4f73b2a050b9ec78c6b1 with peer 2d50f33557944bb480ea32199d9ee4f1
I20240502 21:26:52.050880 1986 tablet_copy_service.cc:342] P 9dec645b473f40578e726f3e50e219bf: Request end of tablet copy session a44f41b0b6644fb6aa29476aed8c964f-acb3bdc518b84b5abc362cc4e12027c3 received from {username='slave'} at 127.24.226.129:41327
I20240502 21:26:52.079331 1986 tablet_copy_service.cc:433] P 9dec645b473f40578e726f3e50e219bf: ending tablet copy session a44f41b0b6644fb6aa29476aed8c964f-acb3bdc518b84b5abc362cc4e12027c3 on tablet acb3bdc518b84b5abc362cc4e12027c3 with peer a44f41b0b6644fb6aa29476aed8c964f
I20240502 21:26:52.056509 3643 consensus_queue.cc:260] T 759b7529d7d84baab1a4622d34f803bf P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:26:52.079999 3643 ts_tablet_manager.cc:1398] T 759b7529d7d84baab1a4622d34f803bf P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.040s user 0.005s sys 0.000s
I20240502 21:26:52.080801 31556 tablet_copy_service.cc:342] P a44f41b0b6644fb6aa29476aed8c964f: Request end of tablet copy session 9dec645b473f40578e726f3e50e219bf-759b7529d7d84baab1a4622d34f803bf received from {username='slave'} at 127.24.226.134:40609
I20240502 21:26:52.080910 31556 tablet_copy_service.cc:433] P a44f41b0b6644fb6aa29476aed8c964f: ending tablet copy session 9dec645b473f40578e726f3e50e219bf-759b7529d7d84baab1a4622d34f803bf on tablet 759b7529d7d84baab1a4622d34f803bf with peer 9dec645b473f40578e726f3e50e219bf
I20240502 21:26:52.081468 3723 tablet_copy_client.cc:647] T b9342dbe0497499ba4c238f2ed033293 P 9dec645b473f40578e726f3e50e219bf: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:52.087751 3723 tablet_copy_client.cc:523] T b9342dbe0497499ba4c238f2ed033293 P 9dec645b473f40578e726f3e50e219bf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:52.090965 3723 tablet_bootstrap.cc:492] T b9342dbe0497499ba4c238f2ed033293 P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:52.133941 3501 raft_consensus.cc:1050] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1: attempting to promote NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b to VOTER
I20240502 21:26:52.134768 3501 consensus_queue.cc:237] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } }
I20240502 21:26:52.136030 3616 tablet_bootstrap.cc:492] T 88189733d9a14560971dbeb71543073f P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:52.136528 3616 tablet_bootstrap.cc:492] T 88189733d9a14560971dbeb71543073f P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap complete.
I20240502 21:26:52.136781 3616 ts_tablet_manager.cc:1367] T 88189733d9a14560971dbeb71543073f P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.225s user 0.041s sys 0.009s
I20240502 21:26:52.137539 3616 raft_consensus.cc:348] T 88189733d9a14560971dbeb71543073f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } }
I20240502 21:26:52.139854 3656 tablet_bootstrap.cc:492] T 53b7eec870c44999b48f0bf744b2ffcd P d1406e92aec84ac585304e57bd37a40d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:52.140269 3656 tablet_bootstrap.cc:492] T 53b7eec870c44999b48f0bf744b2ffcd P d1406e92aec84ac585304e57bd37a40d: Bootstrap complete.
I20240502 21:26:52.141302 31925 consensus_queue.cc:237] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } }
I20240502 21:26:52.137794 3616 raft_consensus.cc:729] T 88189733d9a14560971dbeb71543073f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: LEARNER
I20240502 21:26:52.145236 3616 consensus_queue.cc:260] T 88189733d9a14560971dbeb71543073f P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } }
I20240502 21:26:52.146754 3616 ts_tablet_manager.cc:1398] T 88189733d9a14560971dbeb71543073f P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240502 21:26:52.147393 31947 tablet_copy_service.cc:342] P 2d50f33557944bb480ea32199d9ee4f1: Request end of tablet copy session 1ade0354c44c4c37ae480c7a94b11e1b-88189733d9a14560971dbeb71543073f received from {username='slave'} at 127.24.226.135:33163
I20240502 21:26:52.147504 31947 tablet_copy_service.cc:433] P 2d50f33557944bb480ea32199d9ee4f1: ending tablet copy session 1ade0354c44c4c37ae480c7a94b11e1b-88189733d9a14560971dbeb71543073f on tablet 88189733d9a14560971dbeb71543073f with peer 1ade0354c44c4c37ae480c7a94b11e1b
I20240502 21:26:52.152701 31395 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:26:52.158200 3656 ts_tablet_manager.cc:1367] T 53b7eec870c44999b48f0bf744b2ffcd P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.564s user 0.048s sys 0.005s
I20240502 21:26:52.158953 3656 raft_consensus.cc:348] T 53b7eec870c44999b48f0bf744b2ffcd P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } }
I20240502 21:26:52.159202 3656 raft_consensus.cc:729] T 53b7eec870c44999b48f0bf744b2ffcd P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: LEARNER
I20240502 21:26:52.159421 3656 consensus_queue.cc:260] T 53b7eec870c44999b48f0bf744b2ffcd P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } }
I20240502 21:26:52.166018 3656 ts_tablet_manager.cc:1398] T 53b7eec870c44999b48f0bf744b2ffcd P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240502 21:26:52.173139 31817 tablet_copy_service.cc:342] P 08d7a0680d244b55b93d17f9ab57b84c: Request end of tablet copy session d1406e92aec84ac585304e57bd37a40d-53b7eec870c44999b48f0bf744b2ffcd received from {username='slave'} at 127.24.226.136:57795
I20240502 21:26:52.173266 31817 tablet_copy_service.cc:433] P 08d7a0680d244b55b93d17f9ab57b84c: ending tablet copy session d1406e92aec84ac585304e57bd37a40d-53b7eec870c44999b48f0bf744b2ffcd on tablet 53b7eec870c44999b48f0bf744b2ffcd with peer d1406e92aec84ac585304e57bd37a40d
I20240502 21:26:52.177587 31665 raft_consensus.cc:1261] T 71974e7442b042bebc999b42833be63f P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:52.178156 2230 raft_consensus.cc:1261] T 71974e7442b042bebc999b42833be63f P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:52.178192 3610 consensus_queue.cc:1035] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:52.180037 3610 consensus_queue.cc:1035] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:52.182947 3695 tablet_bootstrap.cc:492] T c5c344f8f84f4a73b853b4f908b93964 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:52.183456 3695 tablet_bootstrap.cc:492] T c5c344f8f84f4a73b853b4f908b93964 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap complete.
I20240502 21:26:52.183795 3695 ts_tablet_manager.cc:1367] T c5c344f8f84f4a73b853b4f908b93964 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.201s user 0.044s sys 0.011s
I20240502 21:26:52.194339 2097 raft_consensus.cc:1261] T dab4f06cc78946dc962267cb2aae04f7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:52.194962 1884 consensus_queue.cc:1035] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:52.200541 3695 raft_consensus.cc:348] T c5c344f8f84f4a73b853b4f908b93964 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } }
I20240502 21:26:52.200796 3695 raft_consensus.cc:729] T c5c344f8f84f4a73b853b4f908b93964 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: LEARNER
I20240502 21:26:52.200991 3695 consensus_queue.cc:260] T c5c344f8f84f4a73b853b4f908b93964 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } }
I20240502 21:26:52.202107 3783 tablet.cc:1722] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:52.205886 31391 catalog_manager.cc:5576] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753) does not have the latest schema for tablet 71974e7442b042bebc999b42833be63f (table rebalancer_dynamic_table [id=9074dd1629f54b0e9607336d3f7224a2]). Expected version 1 got 0
I20240502 21:26:52.207029 3695 ts_tablet_manager.cc:1398] T c5c344f8f84f4a73b853b4f908b93964 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.023s user 0.005s sys 0.000s
I20240502 21:26:52.207286 3783 tablet.cc:1620] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf: MemRowSet was empty: no flush needed.
I20240502 21:26:52.207793 31817 tablet_copy_service.cc:342] P 08d7a0680d244b55b93d17f9ab57b84c: Request end of tablet copy session 1ade0354c44c4c37ae480c7a94b11e1b-c5c344f8f84f4a73b853b4f908b93964 received from {username='slave'} at 127.24.226.135:48019
I20240502 21:26:52.207919 31817 tablet_copy_service.cc:433] P 08d7a0680d244b55b93d17f9ab57b84c: ending tablet copy session 1ade0354c44c4c37ae480c7a94b11e1b-c5c344f8f84f4a73b853b4f908b93964 on tablet c5c344f8f84f4a73b853b4f908b93964 with peer 1ade0354c44c4c37ae480c7a94b11e1b
I20240502 21:26:52.209102 31395 catalog_manager.cc:5576] TS d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907) does not have the latest schema for tablet 71974e7442b042bebc999b42833be63f (table rebalancer_dynamic_table [id=9074dd1629f54b0e9607336d3f7224a2]). Expected version 1 got 0
W20240502 21:26:52.214500 31863 consensus_peers.cc:485] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 -> Peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Couldn't send request to peer a44f41b0b6644fb6aa29476aed8c964f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2db9f537acba4424bca97ffc05c6d0b4. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:52.216747 1833 raft_consensus.cc:1261] T 2db9f537acba4424bca97ffc05c6d0b4 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:52.217434 1884 consensus_queue.cc:1035] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:52.221494 3784 tablet.cc:1722] T 71974e7442b042bebc999b42833be63f P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:52.222731 31395 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=9074dd1629f54b0e9607336d3f7224a2] alter complete (version 1)
I20240502 21:26:52.223294 2095 raft_consensus.cc:1203] T 88189733d9a14560971dbeb71543073f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:26:52.224805 3784 tablet.cc:1620] T 71974e7442b042bebc999b42833be63f P d1406e92aec84ac585304e57bd37a40d: MemRowSet was empty: no flush needed.
I20240502 21:26:52.225555 3726 tablet_bootstrap.cc:492] T 5b1dcae186f542cea92c9191fa8ef951 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:52.226469 3726 tablet_bootstrap.cc:492] T 5b1dcae186f542cea92c9191fa8ef951 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap complete.
I20240502 21:26:52.228782 31924 raft_consensus.cc:1203] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:26:52.231664 3726 ts_tablet_manager.cc:1367] T 5b1dcae186f542cea92c9191fa8ef951 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.203s user 0.045s sys 0.029s
I20240502 21:26:52.232316 3656 ts_tablet_manager.cc:902] T 8ab82de23912483ea178376a5ba5cdc1 P d1406e92aec84ac585304e57bd37a40d: Initiating tablet copy from peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:52.232571 3726 raft_consensus.cc:348] T 5b1dcae186f542cea92c9191fa8ef951 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } }
I20240502 21:26:52.232966 3726 raft_consensus.cc:729] T 5b1dcae186f542cea92c9191fa8ef951 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: LEARNER
I20240502 21:26:52.233116 3504 raft_consensus.cc:2932] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } } }
I20240502 21:26:52.233289 3726 consensus_queue.cc:260] T 5b1dcae186f542cea92c9191fa8ef951 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } }
I20240502 21:26:52.235730 3726 ts_tablet_manager.cc:1398] T 5b1dcae186f542cea92c9191fa8ef951 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:26:52.236536 31817 tablet_copy_service.cc:342] P 08d7a0680d244b55b93d17f9ab57b84c: Request end of tablet copy session 1e04157b98dd4fad98858c290b73c3db-5b1dcae186f542cea92c9191fa8ef951 received from {username='slave'} at 127.24.226.133:41593
I20240502 21:26:52.236686 31817 tablet_copy_service.cc:433] P 08d7a0680d244b55b93d17f9ab57b84c: ending tablet copy session 1e04157b98dd4fad98858c290b73c3db-5b1dcae186f542cea92c9191fa8ef951 on tablet 5b1dcae186f542cea92c9191fa8ef951 with peer 1e04157b98dd4fad98858c290b73c3db
I20240502 21:26:52.241029 1833 raft_consensus.cc:2932] T 2db9f537acba4424bca97ffc05c6d0b4 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } } }
I20240502 21:26:52.243490 3529 consensus_queue.cc:1035] T 2e4ed0121a404869b6fa33b14d3c49f8 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:52.247560 3529 consensus_queue.cc:1035] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:52.248730 3529 consensus_queue.cc:1035] T 2e4ed0121a404869b6fa33b14d3c49f8 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:52.256685 31395 catalog_manager.cc:5576] TS d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907) does not have the latest schema for tablet 71974e7442b042bebc999b42833be63f (table rebalancer_dynamic_table [id=9074dd1629f54b0e9607336d3f7224a2]). Expected version 1 got 0
I20240502 21:26:52.263229 31535 raft_consensus.cc:1203] T acb3bdc518b84b5abc362cc4e12027c3 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:26:52.266379 31391 catalog_manager.cc:5484] T 2db9f537acba4424bca97ffc05c6d0b4 P 1e04157b98dd4fad98858c290b73c3db reported cstate change: config changed from index -1 to 2, NON_VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) added. New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } } }
I20240502 21:26:52.282691 2095 raft_consensus.cc:1261] T 2db9f537acba4424bca97ffc05c6d0b4 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:52.284914 3504 consensus_queue.cc:1035] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:52.289160 2227 raft_consensus.cc:1261] T dab4f06cc78946dc962267cb2aae04f7 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:52.289918 1884 consensus_queue.cc:1035] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:52.292627 3529 consensus_queue.cc:1035] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:52.298835 2095 raft_consensus.cc:2932] T 2db9f537acba4424bca97ffc05c6d0b4 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } } }
I20240502 21:26:52.306310 1833 raft_consensus.cc:1261] T dab4f06cc78946dc962267cb2aae04f7 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:52.307106 3504 consensus_queue.cc:1035] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:52.320480 31391 catalog_manager.cc:5576] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645) does not have the latest schema for tablet 71974e7442b042bebc999b42833be63f (table rebalancer_dynamic_table [id=9074dd1629f54b0e9607336d3f7224a2]). Expected version 1 got 0
I20240502 21:26:52.332530 3656 tablet_copy_client.cc:308] T 8ab82de23912483ea178376a5ba5cdc1 P d1406e92aec84ac585304e57bd37a40d: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.129:41039
I20240502 21:26:52.333271 31556 tablet_copy_service.cc:140] P a44f41b0b6644fb6aa29476aed8c964f: Received BeginTabletCopySession request for tablet 8ab82de23912483ea178376a5ba5cdc1 from peer d1406e92aec84ac585304e57bd37a40d ({username='slave'} at 127.24.226.136:52831)
I20240502 21:26:52.333446 31556 tablet_copy_service.cc:161] P a44f41b0b6644fb6aa29476aed8c964f: Beginning new tablet copy session on tablet 8ab82de23912483ea178376a5ba5cdc1 from peer d1406e92aec84ac585304e57bd37a40d at {username='slave'} at 127.24.226.136:52831: session id = d1406e92aec84ac585304e57bd37a40d-8ab82de23912483ea178376a5ba5cdc1
I20240502 21:26:52.332720 3504 raft_consensus.cc:1050] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1: attempting to promote NON_VOTER 9dec645b473f40578e726f3e50e219bf to VOTER
I20240502 21:26:52.334589 3504 consensus_queue.cc:237] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:26:52.336833 31556 tablet_copy_source_session.cc:205] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:52.337816 31391 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:26:52.338413 3656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab82de23912483ea178376a5ba5cdc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:52.347254 3656 tablet_copy_client.cc:783] T 8ab82de23912483ea178376a5ba5cdc1 P d1406e92aec84ac585304e57bd37a40d: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:52.347470 3656 tablet_copy_client.cc:647] T 8ab82de23912483ea178376a5ba5cdc1 P d1406e92aec84ac585304e57bd37a40d: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:52.349603 3656 tablet_copy_client.cc:523] T 8ab82de23912483ea178376a5ba5cdc1 P d1406e92aec84ac585304e57bd37a40d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:52.352805 3783 tablet.cc:1722] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:52.353425 3783 tablet.cc:1620] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf: MemRowSet was empty: no flush needed.
I20240502 21:26:52.358114 3810 tablet.cc:1722] T 71974e7442b042bebc999b42833be63f P 05f4fac41c4f4959b9b316cb112d0ada: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:52.363034 31395 catalog_manager.cc:5576] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645) does not have the latest schema for tablet 71974e7442b042bebc999b42833be63f (table rebalancer_dynamic_table [id=9074dd1629f54b0e9607336d3f7224a2]). Expected version 2 got 0
I20240502 21:26:52.367945 31394 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=9074dd1629f54b0e9607336d3f7224a2] alter complete (version 2)
I20240502 21:26:52.371176 3644 tablet_bootstrap.cc:492] T ae56d18c629a4e5eb50911c89de4227c P 9dec645b473f40578e726f3e50e219bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:52.371604 3644 tablet_bootstrap.cc:492] T ae56d18c629a4e5eb50911c89de4227c P 9dec645b473f40578e726f3e50e219bf: Bootstrap complete.
I20240502 21:26:52.374100 1965 raft_consensus.cc:1261] T 62f083e9641042c990efcfb1c4104de5 P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:26:52.380596 3810 tablet.cc:1620] T 71974e7442b042bebc999b42833be63f P 05f4fac41c4f4959b9b316cb112d0ada: MemRowSet was empty: no flush needed.
I20240502 21:26:52.385244 3656 tablet_bootstrap.cc:492] T 8ab82de23912483ea178376a5ba5cdc1 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:52.385154 3529 consensus_queue.cc:1035] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:52.388382 3784 tablet.cc:1722] T 71974e7442b042bebc999b42833be63f P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:52.388804 3784 tablet.cc:1620] T 71974e7442b042bebc999b42833be63f P d1406e92aec84ac585304e57bd37a40d: MemRowSet was empty: no flush needed.
I20240502 21:26:52.400451 3501 consensus_queue.cc:1035] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:26:52.405867 2227 consensus_queue.cc:237] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } }
W20240502 21:26:52.418216 2163 consensus_peers.cc:485] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d -> Peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): Couldn't send request to peer 08d7a0680d244b55b93d17f9ab57b84c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f5e82c0ab5c441efbcfbcd383b8db82f. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:52.419461 2097 raft_consensus.cc:1261] T f5e82c0ab5c441efbcfbcd383b8db82f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:52.420061 3769 consensus_queue.cc:1035] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:52.425499 1833 raft_consensus.cc:1261] T f5e82c0ab5c441efbcfbcd383b8db82f P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:52.425549 31394 catalog_manager.cc:5576] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645) does not have the latest schema for tablet 71974e7442b042bebc999b42833be63f (table rebalancer_dynamic_table [id=9074dd1629f54b0e9607336d3f7224a2]). Expected version 2 got 1
I20240502 21:26:52.426067 3769 consensus_queue.cc:1035] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:52.430579 3828 tablet.cc:1722] T 71974e7442b042bebc999b42833be63f P 05f4fac41c4f4959b9b316cb112d0ada: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:52.432703 3828 tablet.cc:1620] T 71974e7442b042bebc999b42833be63f P 05f4fac41c4f4959b9b316cb112d0ada: MemRowSet was empty: no flush needed.
I20240502 21:26:52.435030 3669 consensus_queue.cc:1035] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:52.441040 3644 ts_tablet_manager.cc:1367] T ae56d18c629a4e5eb50911c89de4227c P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.458s user 0.045s sys 0.007s
I20240502 21:26:52.442442 3723 tablet_bootstrap.cc:492] T b9342dbe0497499ba4c238f2ed033293 P 9dec645b473f40578e726f3e50e219bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:52.442144 3769 raft_consensus.cc:2932] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } } }
I20240502 21:26:52.442422 3813 raft_consensus.cc:2932] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
W20240502 21:26:52.444486 31732 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.226.133:39863, user_credentials={real_user=slave}} blocked reactor thread for 120686us
I20240502 21:26:52.445801 1833 raft_consensus.cc:1203] T 5b1dcae186f542cea92c9191fa8ef951 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:26:52.444263 3644 raft_consensus.cc:348] T ae56d18c629a4e5eb50911c89de4227c P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:26:52.446790 3644 raft_consensus.cc:729] T ae56d18c629a4e5eb50911c89de4227c P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: LEARNER
I20240502 21:26:52.447032 3644 consensus_queue.cc:260] T ae56d18c629a4e5eb50911c89de4227c P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:26:52.447314 3723 tablet_bootstrap.cc:492] T b9342dbe0497499ba4c238f2ed033293 P 9dec645b473f40578e726f3e50e219bf: Bootstrap complete.
I20240502 21:26:52.447552 3723 ts_tablet_manager.cc:1367] T b9342dbe0497499ba4c238f2ed033293 P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.357s user 0.010s sys 0.002s
I20240502 21:26:52.449952 3644 ts_tablet_manager.cc:1398] T ae56d18c629a4e5eb50911c89de4227c P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 21:26:52.452950 31391 catalog_manager.cc:5484] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d reported cstate change: config changed from index -1 to 2, NON_VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) added. New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:52.454849 1833 raft_consensus.cc:2932] T f5e82c0ab5c441efbcfbcd383b8db82f P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } } }
I20240502 21:26:52.450356 3723 raft_consensus.cc:348] T b9342dbe0497499ba4c238f2ed033293 P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:26:52.457852 3723 raft_consensus.cc:729] T b9342dbe0497499ba4c238f2ed033293 P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: LEARNER
I20240502 21:26:52.458112 3723 consensus_queue.cc:260] T b9342dbe0497499ba4c238f2ed033293 P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:26:52.461611 3723 ts_tablet_manager.cc:1398] T b9342dbe0497499ba4c238f2ed033293 P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20240502 21:26:52.468340 1854 tablet_copy_service.cc:342] P 1e04157b98dd4fad98858c290b73c3db: Request end of tablet copy session 9dec645b473f40578e726f3e50e219bf-b9342dbe0497499ba4c238f2ed033293 received from {username='slave'} at 127.24.226.134:51927
I20240502 21:26:52.468485 1854 tablet_copy_service.cc:433] P 1e04157b98dd4fad98858c290b73c3db: ending tablet copy session 9dec645b473f40578e726f3e50e219bf-b9342dbe0497499ba4c238f2ed033293 on tablet b9342dbe0497499ba4c238f2ed033293 with peer 9dec645b473f40578e726f3e50e219bf
W20240502 21:26:52.473290 31860 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.226.134:43753, user_credentials={real_user=slave}} blocked reactor thread for 98647.7us
I20240502 21:26:52.474412 1965 raft_consensus.cc:1203] T ae56d18c629a4e5eb50911c89de4227c P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:26:52.476615 3671 tablet_bootstrap.cc:492] T 5a54d2aad63c4f3182992f00b8564acf P d1406e92aec84ac585304e57bd37a40d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20240502 21:26:52.476850 31862 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.226.135:38569, user_credentials={real_user=slave}} blocked reactor thread for 86647.4us
I20240502 21:26:52.477480 2227 raft_consensus.cc:1261] T 62f083e9641042c990efcfb1c4104de5 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:26:52.477841 31394 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:26:52.478245 3671 tablet_bootstrap.cc:492] T 5a54d2aad63c4f3182992f00b8564acf P d1406e92aec84ac585304e57bd37a40d: Bootstrap complete.
I20240502 21:26:52.478562 3671 ts_tablet_manager.cc:1367] T 5a54d2aad63c4f3182992f00b8564acf P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.509s user 0.057s sys 0.000s
I20240502 21:26:52.479341 3671 raft_consensus.cc:348] T 5a54d2aad63c4f3182992f00b8564acf P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } }
I20240502 21:26:52.479677 3671 raft_consensus.cc:729] T 5a54d2aad63c4f3182992f00b8564acf P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: LEARNER
I20240502 21:26:52.479897 3671 consensus_queue.cc:260] T 5a54d2aad63c4f3182992f00b8564acf P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } }
I20240502 21:26:52.483268 3671 ts_tablet_manager.cc:1398] T 5a54d2aad63c4f3182992f00b8564acf P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:26:52.484649 31817 tablet_copy_service.cc:342] P 08d7a0680d244b55b93d17f9ab57b84c: Request end of tablet copy session d1406e92aec84ac585304e57bd37a40d-5a54d2aad63c4f3182992f00b8564acf received from {username='slave'} at 127.24.226.136:57795
I20240502 21:26:52.484771 31817 tablet_copy_service.cc:433] P 08d7a0680d244b55b93d17f9ab57b84c: ending tablet copy session d1406e92aec84ac585304e57bd37a40d-5a54d2aad63c4f3182992f00b8564acf on tablet 5a54d2aad63c4f3182992f00b8564acf with peer d1406e92aec84ac585304e57bd37a40d
I20240502 21:26:52.493558 3783 tablet.cc:1722] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:52.494230 2095 raft_consensus.cc:2932] T dab4f06cc78946dc962267cb2aae04f7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:52.495302 3501 consensus_queue.cc:1035] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:26:52.496021 31796 raft_consensus.cc:1261] T 62f083e9641042c990efcfb1c4104de5 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:26:52.497500 3784 tablet.cc:1722] T 71974e7442b042bebc999b42833be63f P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:52.498423 3828 tablet.cc:1722] T 71974e7442b042bebc999b42833be63f P 05f4fac41c4f4959b9b316cb112d0ada: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:52.499948 31394 catalog_manager.cc:5484] T dab4f06cc78946dc962267cb2aae04f7 P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: config changed from index 2 to 3, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:52.505632 31391 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_extra_table_20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:52.507519 31390 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table_new_name [id=9074dd1629f54b0e9607336d3f7224a2] alter complete (version 3)
I20240502 21:26:52.518896 3656 tablet_bootstrap.cc:492] T 8ab82de23912483ea178376a5ba5cdc1 P d1406e92aec84ac585304e57bd37a40d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:52.519363 3656 tablet_bootstrap.cc:492] T 8ab82de23912483ea178376a5ba5cdc1 P d1406e92aec84ac585304e57bd37a40d: Bootstrap complete.
I20240502 21:26:52.519650 3656 ts_tablet_manager.cc:1367] T 8ab82de23912483ea178376a5ba5cdc1 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.134s user 0.033s sys 0.016s
I20240502 21:26:52.519007 2096 raft_consensus.cc:2932] T f5e82c0ab5c441efbcfbcd383b8db82f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } } }
I20240502 21:26:52.521034 3656 raft_consensus.cc:348] T 8ab82de23912483ea178376a5ba5cdc1 P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } }
I20240502 21:26:52.521236 3656 raft_consensus.cc:729] T 8ab82de23912483ea178376a5ba5cdc1 P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: LEARNER
I20240502 21:26:52.521445 3656 consensus_queue.cc:260] T 8ab82de23912483ea178376a5ba5cdc1 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } }
I20240502 21:26:52.522979 3656 ts_tablet_manager.cc:1398] T 8ab82de23912483ea178376a5ba5cdc1 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20240502 21:26:52.523566 31556 tablet_copy_service.cc:342] P a44f41b0b6644fb6aa29476aed8c964f: Request end of tablet copy session d1406e92aec84ac585304e57bd37a40d-8ab82de23912483ea178376a5ba5cdc1 received from {username='slave'} at 127.24.226.136:52831
I20240502 21:26:52.523698 31556 tablet_copy_service.cc:433] P a44f41b0b6644fb6aa29476aed8c964f: ending tablet copy session d1406e92aec84ac585304e57bd37a40d-8ab82de23912483ea178376a5ba5cdc1 on tablet 8ab82de23912483ea178376a5ba5cdc1 with peer d1406e92aec84ac585304e57bd37a40d
I20240502 21:26:52.524700 31645 tablet_service.cc:1467] Processing CreateTablet for tablet d5f7ee5a6c8a41fc80a5ef1c65a75579 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=aae794f4f92e4a4b9f2610d66d67525d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:52.525105 31645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5f7ee5a6c8a41fc80a5ef1c65a75579. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:52.525308 31947 tablet_copy_service.cc:342] P 2d50f33557944bb480ea32199d9ee4f1: Request end of tablet copy session 9dec645b473f40578e726f3e50e219bf-ae56d18c629a4e5eb50911c89de4227c received from {username='slave'} at 127.24.226.134:56721
I20240502 21:26:52.525414 31947 tablet_copy_service.cc:433] P 2d50f33557944bb480ea32199d9ee4f1: ending tablet copy session 9dec645b473f40578e726f3e50e219bf-ae56d18c629a4e5eb50911c89de4227c on tablet ae56d18c629a4e5eb50911c89de4227c with peer 9dec645b473f40578e726f3e50e219bf
I20240502 21:26:52.527422 1833 raft_consensus.cc:2932] T dab4f06cc78946dc962267cb2aae04f7 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:52.531176 1945 tablet_service.cc:1467] Processing CreateTablet for tablet 2068521c26dc44f9ba60e274eb039b22 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=aae794f4f92e4a4b9f2610d66d67525d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:52.531519 1945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2068521c26dc44f9ba60e274eb039b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:52.538807 31775 tablet_service.cc:1467] Processing CreateTablet for tablet 534b4d96df9b4bc5a9480b6c45eb11a5 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=aae794f4f92e4a4b9f2610d66d67525d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:52.538883 2076 tablet_service.cc:1467] Processing CreateTablet for tablet 534b4d96df9b4bc5a9480b6c45eb11a5 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=aae794f4f92e4a4b9f2610d66d67525d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:52.539239 31775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 534b4d96df9b4bc5a9480b6c45eb11a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:52.539251 2076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 534b4d96df9b4bc5a9480b6c45eb11a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:52.539417 2208 tablet_service.cc:1467] Processing CreateTablet for tablet 2068521c26dc44f9ba60e274eb039b22 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=aae794f4f92e4a4b9f2610d66d67525d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:52.539403 2075 tablet_service.cc:1467] Processing CreateTablet for tablet d5f7ee5a6c8a41fc80a5ef1c65a75579 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=aae794f4f92e4a4b9f2610d66d67525d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:52.539755 2208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2068521c26dc44f9ba60e274eb039b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:52.539764 2075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5f7ee5a6c8a41fc80a5ef1c65a75579. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:52.544008 2077 tablet_service.cc:1467] Processing CreateTablet for tablet 2068521c26dc44f9ba60e274eb039b22 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=aae794f4f92e4a4b9f2610d66d67525d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:52.544355 2077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2068521c26dc44f9ba60e274eb039b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:52.544951 1813 tablet_service.cc:1467] Processing CreateTablet for tablet d5f7ee5a6c8a41fc80a5ef1c65a75579 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=aae794f4f92e4a4b9f2610d66d67525d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:52.545475 1813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5f7ee5a6c8a41fc80a5ef1c65a75579. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:52.548827 2210 tablet_service.cc:1467] Processing CreateTablet for tablet 534b4d96df9b4bc5a9480b6c45eb11a5 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=aae794f4f92e4a4b9f2610d66d67525d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:52.549278 2210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 534b4d96df9b4bc5a9480b6c45eb11a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:52.551961 3821 consensus_queue.cc:1035] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:26:52.570448 1966 raft_consensus.cc:1203] T b9342dbe0497499ba4c238f2ed033293 P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 21:26:52.571722 2230 raft_consensus.cc:2932] T dab4f06cc78946dc962267cb2aae04f7 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:52.573634 3845 tablet_bootstrap.cc:492] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:52.582094 31924 consensus_queue.cc:237] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } }
I20240502 21:26:52.585294 2096 raft_consensus.cc:1261] T dab4f06cc78946dc962267cb2aae04f7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:52.586391 3845 tablet_bootstrap.cc:654] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:52.586751 3821 consensus_queue.cc:1035] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:52.588778 1833 raft_consensus.cc:1261] T dab4f06cc78946dc962267cb2aae04f7 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:52.589691 3840 tablet_bootstrap.cc:492] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:52.590451 3821 consensus_queue.cc:1035] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:52.592329 3840 tablet_bootstrap.cc:654] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:52.592710 3504 raft_consensus.cc:2932] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:52.597173 3843 tablet_bootstrap.cc:492] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:52.601397 1833 raft_consensus.cc:2932] T dab4f06cc78946dc962267cb2aae04f7 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:52.601883 31394 catalog_manager.cc:5484] T dab4f06cc78946dc962267cb2aae04f7 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: config changed from index 3 to 4, VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) evicted. New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:52.604583 31382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dab4f06cc78946dc962267cb2aae04f7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:52.605445 31795 consensus_queue.cc:237] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:26:52.607440 3843 tablet_bootstrap.cc:654] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:52.612828 31389 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:52.613080 31389 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:52.613512 3822 raft_consensus.cc:2932] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:52.618288 1966 raft_consensus.cc:2932] T 62f083e9641042c990efcfb1c4104de5 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:52.620947 31393 catalog_manager.cc:5484] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: config changed from index 7 to 8, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:52.623747 31389 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 71974e7442b042bebc999b42833be63f
I20240502 21:26:52.625735 31645 tablet_service.cc:1514] Processing DeleteTablet for tablet 71974e7442b042bebc999b42833be63f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:52 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:52.626178 1946 tablet_service.cc:1514] Processing DeleteTablet for tablet 71974e7442b042bebc999b42833be63f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:52 UTC) from {username='slave'} at 127.0.0.1:33002
I20240502 21:26:52.630381 3846 tablet_bootstrap.cc:492] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:52.634475 2096 raft_consensus.cc:2932] T dab4f06cc78946dc962267cb2aae04f7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:52.620611 2230 raft_consensus.cc:2932] T 62f083e9641042c990efcfb1c4104de5 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:52.642624 1965 raft_consensus.cc:1203] T 759b7529d7d84baab1a4622d34f803bf P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:26:52.642643 31924 consensus_queue.cc:237] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:26:52.645080 31665 raft_consensus.cc:1261] T fc93b5adc1d94e809e97e56b56822611 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:52.645756 1964 raft_consensus.cc:1261] T 62f083e9641042c990efcfb1c4104de5 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:26:52.645979 31536 raft_consensus.cc:1261] T fc93b5adc1d94e809e97e56b56822611 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:26:52.646282 31730 consensus_peers.cc:485] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc93b5adc1d94e809e97e56b56822611. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:52.646658 3822 consensus_queue.cc:1035] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:26:52.647300 3517 consensus_queue.cc:1035] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:52.647873 3812 consensus_queue.cc:1035] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:52.649839 2230 raft_consensus.cc:1261] T 62f083e9641042c990efcfb1c4104de5 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:26:52.621225 2208 tablet_service.cc:1514] Processing DeleteTablet for tablet dab4f06cc78946dc962267cb2aae04f7 with delete_type TABLET_DATA_TOMBSTONED (TS d1406e92aec84ac585304e57bd37a40d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36984
I20240502 21:26:52.625691 2210 tablet_service.cc:1514] Processing DeleteTablet for tablet 71974e7442b042bebc999b42833be63f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:52 UTC) from {username='slave'} at 127.0.0.1:36984
I20240502 21:26:52.633863 2227 raft_consensus.cc:1203] T 53b7eec870c44999b48f0bf744b2ffcd P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:26:52.652292 3504 consensus_queue.cc:1035] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:26:52.656371 2631 raft_consensus.cc:1050] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f: attempting to promote NON_VOTER 2d50f33557944bb480ea32199d9ee4f1 to VOTER
I20240502 21:26:52.656059 3813 raft_consensus.cc:2932] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:52.657546 2631 consensus_queue.cc:237] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } }
I20240502 21:26:52.660873 1966 raft_consensus.cc:2932] T 62f083e9641042c990efcfb1c4104de5 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:52.665510 2227 raft_consensus.cc:2932] T 62f083e9641042c990efcfb1c4104de5 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:52.671887 31393 catalog_manager.cc:5484] T 62f083e9641042c990efcfb1c4104de5 P d1406e92aec84ac585304e57bd37a40d reported cstate change: config changed from index 8 to 9, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:52.673141 3841 tablet_bootstrap.cc:492] T 2068521c26dc44f9ba60e274eb039b22 P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:52.676147 3842 tablet_bootstrap.cc:492] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:52.677819 3842 tablet_bootstrap.cc:654] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:52.678385 31775 tablet_service.cc:1514] Processing DeleteTablet for tablet 62f083e9641042c990efcfb1c4104de5 with delete_type TABLET_DATA_TOMBSTONED (TS 08d7a0680d244b55b93d17f9ab57b84c not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51596
I20240502 21:26:52.680061 3840 tablet_bootstrap.cc:492] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:52.680248 3846 tablet_bootstrap.cc:654] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:52.684319 3840 ts_tablet_manager.cc:1367] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.095s user 0.002s sys 0.002s
I20240502 21:26:52.690665 3841 tablet_bootstrap.cc:654] T 2068521c26dc44f9ba60e274eb039b22 P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:52.690659 3610 raft_consensus.cc:1050] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf: attempting to promote NON_VOTER a44f41b0b6644fb6aa29476aed8c964f to VOTER
I20240502 21:26:52.691116 3851 tablet_replica.cc:330] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf: stopping tablet replica
I20240502 21:26:52.693850 3851 raft_consensus.cc:2229] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:52.694259 3851 raft_consensus.cc:2258] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:52.694383 3610 consensus_queue.cc:237] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } }
I20240502 21:26:52.695209 3851 ts_tablet_manager.cc:1875] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:52.695516 3857 tablet_replica.cc:330] T 62f083e9641042c990efcfb1c4104de5 P 08d7a0680d244b55b93d17f9ab57b84c: stopping tablet replica
I20240502 21:26:52.695792 3857 raft_consensus.cc:2229] T 62f083e9641042c990efcfb1c4104de5 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:52.696046 3857 pending_rounds.cc:62] T 62f083e9641042c990efcfb1c4104de5 P 08d7a0680d244b55b93d17f9ab57b84c: Trying to abort 1 pending ops.
I20240502 21:26:52.696228 3857 pending_rounds.cc:68] T 62f083e9641042c990efcfb1c4104de5 P 08d7a0680d244b55b93d17f9ab57b84c: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7023350629720547328 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "62f083e9641042c990efcfb1c4104de5" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } } }
I20240502 21:26:52.696913 3857 raft_consensus.cc:2894] T 62f083e9641042c990efcfb1c4104de5 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240502 21:26:52.697110 3857 raft_consensus.cc:2866] T 62f083e9641042c990efcfb1c4104de5 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 21:26:52.697299 3857 raft_consensus.cc:2258] T 62f083e9641042c990efcfb1c4104de5 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:52.698220 3857 ts_tablet_manager.cc:1875] T 62f083e9641042c990efcfb1c4104de5 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:52.702742 3846 tablet_bootstrap.cc:492] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:52.699158 3840 raft_consensus.cc:348] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:52.707259 3840 raft_consensus.cc:374] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:52.708061 3840 raft_consensus.cc:729] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:52.709208 3842 tablet_bootstrap.cc:492] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:52.709555 3840 consensus_queue.cc:260] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:52.708992 3857 ts_tablet_manager.cc:1888] T 62f083e9641042c990efcfb1c4104de5 P 08d7a0680d244b55b93d17f9ab57b84c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:26:52.712801 3846 ts_tablet_manager.cc:1367] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.083s user 0.002s sys 0.002s
I20240502 21:26:52.712842 3857 log.cc:1196] T 62f083e9641042c990efcfb1c4104de5 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/wals/62f083e9641042c990efcfb1c4104de5
I20240502 21:26:52.714313 31381 catalog_manager.cc:4830] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): tablet 62f083e9641042c990efcfb1c4104de5 (table rebalance_test_table_17 [id=2e8e8260e3b64486b5712ad7289d1cc8]) successfully deleted
I20240502 21:26:52.716303 3841 tablet_bootstrap.cc:492] T 2068521c26dc44f9ba60e274eb039b22 P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:26:52.716939 3845 tablet_bootstrap.cc:492] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:52.717286 3845 ts_tablet_manager.cc:1367] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.144s user 0.004s sys 0.003s
I20240502 21:26:52.718487 3845 raft_consensus.cc:348] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:52.718880 3845 raft_consensus.cc:374] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:52.719033 3845 raft_consensus.cc:729] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:52.719343 3845 consensus_queue.cc:260] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:52.725190 31382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 62f083e9641042c990efcfb1c4104de5 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:52.730443 31395 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:26:52.734477 3708 tablet_bootstrap.cc:492] T 55fc18e4cc98446ca50c1dd2b6a9a89d P d1406e92aec84ac585304e57bd37a40d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:52.736619 3708 tablet_bootstrap.cc:492] T 55fc18e4cc98446ca50c1dd2b6a9a89d P d1406e92aec84ac585304e57bd37a40d: Bootstrap complete.
I20240502 21:26:52.736908 2230 raft_consensus.cc:1203] T 5a54d2aad63c4f3182992f00b8564acf P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:26:52.737962 3708 ts_tablet_manager.cc:1367] T 55fc18e4cc98446ca50c1dd2b6a9a89d P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.711s user 0.055s sys 0.000s
I20240502 21:26:52.740880 3840 ts_tablet_manager.cc:1398] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.055s user 0.001s sys 0.001s
I20240502 21:26:52.741194 3842 ts_tablet_manager.cc:1367] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.065s user 0.002s sys 0.001s
I20240502 21:26:52.741925 3842 raft_consensus.cc:348] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:52.742141 3842 raft_consensus.cc:374] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:52.742244 3842 raft_consensus.cc:729] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:52.742460 3842 consensus_queue.cc:260] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:52.744611 3841 ts_tablet_manager.cc:1367] T 2068521c26dc44f9ba60e274eb039b22 P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.072s user 0.004s sys 0.000s
I20240502 21:26:52.748901 3846 raft_consensus.cc:348] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:52.749835 3840 tablet_bootstrap.cc:492] T 534b4d96df9b4bc5a9480b6c45eb11a5 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:52.750047 3841 raft_consensus.cc:348] T 2068521c26dc44f9ba60e274eb039b22 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:52.750238 3841 raft_consensus.cc:374] T 2068521c26dc44f9ba60e274eb039b22 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:52.750334 3841 raft_consensus.cc:729] T 2068521c26dc44f9ba60e274eb039b22 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:26:52.749943 3708 raft_consensus.cc:348] T 55fc18e4cc98446ca50c1dd2b6a9a89d P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } }
I20240502 21:26:52.750620 3708 raft_consensus.cc:729] T 55fc18e4cc98446ca50c1dd2b6a9a89d P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: LEARNER
I20240502 21:26:52.751492 3843 tablet_bootstrap.cc:492] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:52.751634 3843 ts_tablet_manager.cc:1367] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.155s user 0.004s sys 0.022s
I20240502 21:26:52.752174 3824 raft_consensus.cc:1050] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1: attempting to promote NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b to VOTER
I20240502 21:26:52.752467 3843 raft_consensus.cc:348] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:52.752684 3843 raft_consensus.cc:374] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:52.752772 3843 raft_consensus.cc:729] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:52.753062 3843 consensus_queue.cc:260] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:52.754909 3843 ts_tablet_manager.cc:1398] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:26:52.755374 3846 raft_consensus.cc:374] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:52.755474 3846 raft_consensus.cc:729] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:52.755618 3824 consensus_queue.cc:237] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } }
I20240502 21:26:52.756018 3651 tablet_bootstrap.cc:492] T a1fd14de204c40d98027ecb329ff79f3 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:52.756386 3651 tablet_bootstrap.cc:492] T a1fd14de204c40d98027ecb329ff79f3 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap complete.
I20240502 21:26:52.756677 3651 ts_tablet_manager.cc:1367] T a1fd14de204c40d98027ecb329ff79f3 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 1.020s user 0.044s sys 0.020s
I20240502 21:26:52.757939 3846 consensus_queue.cc:260] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:52.760216 3708 consensus_queue.cc:260] T 55fc18e4cc98446ca50c1dd2b6a9a89d P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } }
I20240502 21:26:52.760418 31536 raft_consensus.cc:1261] T 88189733d9a14560971dbeb71543073f P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:52.761312 3504 consensus_queue.cc:1035] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:52.764703 3846 ts_tablet_manager.cc:1398] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.051s user 0.001s sys 0.001s
I20240502 21:26:52.765681 3842 ts_tablet_manager.cc:1398] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20240502 21:26:52.766763 3708 ts_tablet_manager.cc:1398] T 55fc18e4cc98446ca50c1dd2b6a9a89d P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20240502 21:26:52.765003 3841 consensus_queue.cc:260] T 2068521c26dc44f9ba60e274eb039b22 P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:52.767668 3842 tablet_bootstrap.cc:492] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:52.766595 2095 raft_consensus.cc:1261] T 88189733d9a14560971dbeb71543073f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:52.770190 31556 tablet_copy_service.cc:342] P a44f41b0b6644fb6aa29476aed8c964f: Request end of tablet copy session d1406e92aec84ac585304e57bd37a40d-55fc18e4cc98446ca50c1dd2b6a9a89d received from {username='slave'} at 127.24.226.136:52831
I20240502 21:26:52.770318 31556 tablet_copy_service.cc:433] P a44f41b0b6644fb6aa29476aed8c964f: ending tablet copy session d1406e92aec84ac585304e57bd37a40d-55fc18e4cc98446ca50c1dd2b6a9a89d on tablet 55fc18e4cc98446ca50c1dd2b6a9a89d with peer d1406e92aec84ac585304e57bd37a40d
I20240502 21:26:52.770365 3841 ts_tablet_manager.cc:1398] T 2068521c26dc44f9ba60e274eb039b22 P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240502 21:26:52.757390 3651 raft_consensus.cc:348] T a1fd14de204c40d98027ecb329ff79f3 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } }
I20240502 21:26:52.774466 3651 raft_consensus.cc:729] T a1fd14de204c40d98027ecb329ff79f3 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: LEARNER
I20240502 21:26:52.774693 3651 consensus_queue.cc:260] T a1fd14de204c40d98027ecb329ff79f3 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } }
I20240502 21:26:52.775692 3840 tablet_bootstrap.cc:654] T 534b4d96df9b4bc5a9480b6c45eb11a5 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:52.778403 3651 ts_tablet_manager.cc:1398] T a1fd14de204c40d98027ecb329ff79f3 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.022s user 0.000s sys 0.002s
I20240502 21:26:52.779004 31817 tablet_copy_service.cc:342] P 08d7a0680d244b55b93d17f9ab57b84c: Request end of tablet copy session 1e04157b98dd4fad98858c290b73c3db-a1fd14de204c40d98027ecb329ff79f3 received from {username='slave'} at 127.24.226.133:41593
I20240502 21:26:52.779114 31817 tablet_copy_service.cc:433] P 08d7a0680d244b55b93d17f9ab57b84c: ending tablet copy session 1e04157b98dd4fad98858c290b73c3db-a1fd14de204c40d98027ecb329ff79f3 on tablet a1fd14de204c40d98027ecb329ff79f3 with peer 1e04157b98dd4fad98858c290b73c3db
I20240502 21:26:52.782120 3842 tablet_bootstrap.cc:654] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:52.785239 2095 raft_consensus.cc:1203] T c5c344f8f84f4a73b853b4f908b93964 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:26:52.785514 3851 ts_tablet_manager.cc:1888] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:52.785631 3851 log.cc:1196] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal/wals/71974e7442b042bebc999b42833be63f
I20240502 21:26:52.792989 3812 raft_consensus.cc:482] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:52.793186 3812 raft_consensus.cc:504] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:52.793678 3868 ts_tablet_manager.cc:902] T 2db9f537acba4424bca97ffc05c6d0b4 P a44f41b0b6644fb6aa29476aed8c964f: Initiating tablet copy from peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:52.793879 3851 ts_tablet_manager.cc:1909] T 71974e7442b042bebc999b42833be63f P 9dec645b473f40578e726f3e50e219bf: Deleting consensus metadata
I20240502 21:26:52.794118 3812 leader_election.cc:288] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:52.795238 3842 tablet_bootstrap.cc:492] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:52.795158 2230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534b4d96df9b4bc5a9480b6c45eb11a5" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:26:52.795593 3842 ts_tablet_manager.cc:1367] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20240502 21:26:52.795831 31379 catalog_manager.cc:4830] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): tablet 71974e7442b042bebc999b42833be63f (table rebalancer_dynamic_table_new_name [id=9074dd1629f54b0e9607336d3f7224a2]) successfully deleted
W20240502 21:26:52.796135 31732 leader_election.cc:341] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:52.796559 2095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534b4d96df9b4bc5a9480b6c45eb11a5" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:52.796823 2095 raft_consensus.cc:2454] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:52.797395 31732 leader_election.cc:302] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 1ade0354c44c4c37ae480c7a94b11e1b; no voters: d1406e92aec84ac585304e57bd37a40d
I20240502 21:26:52.798255 3812 raft_consensus.cc:2781] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:52.798377 3812 raft_consensus.cc:482] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:52.798486 3812 raft_consensus.cc:3037] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:52.803295 3842 raft_consensus.cc:348] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:52.805095 3812 raft_consensus.cc:504] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:52.805444 3842 raft_consensus.cc:374] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:52.805783 3842 raft_consensus.cc:729] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:52.806231 3842 consensus_queue.cc:260] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:52.808756 2230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534b4d96df9b4bc5a9480b6c45eb11a5" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:26:52.808897 3812 leader_election.cc:288] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Requested vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:52.809701 2230 raft_consensus.cc:1203] T 8ab82de23912483ea178376a5ba5cdc1 P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:26:52.807611 2095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534b4d96df9b4bc5a9480b6c45eb11a5" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:26:52.810293 2095 raft_consensus.cc:3037] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:52.810536 3842 ts_tablet_manager.cc:1398] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240502 21:26:52.810806 3842 tablet_bootstrap.cc:492] T 2068521c26dc44f9ba60e274eb039b22 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:52.812383 3842 tablet_bootstrap.cc:654] T 2068521c26dc44f9ba60e274eb039b22 P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
W20240502 21:26:52.815846 31732 leader_election.cc:341] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:52.817318 3840 tablet_bootstrap.cc:492] T 534b4d96df9b4bc5a9480b6c45eb11a5 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:52.817549 3840 ts_tablet_manager.cc:1367] T 534b4d96df9b4bc5a9480b6c45eb11a5 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.068s user 0.007s sys 0.000s
I20240502 21:26:52.818198 3840 raft_consensus.cc:348] T 534b4d96df9b4bc5a9480b6c45eb11a5 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:52.818565 3840 raft_consensus.cc:374] T 534b4d96df9b4bc5a9480b6c45eb11a5 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:52.818658 3840 raft_consensus.cc:729] T 534b4d96df9b4bc5a9480b6c45eb11a5 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:52.818930 3840 consensus_queue.cc:260] T 534b4d96df9b4bc5a9480b6c45eb11a5 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:52.819648 2095 raft_consensus.cc:2454] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:52.822714 31732 leader_election.cc:302] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 1ade0354c44c4c37ae480c7a94b11e1b; no voters: d1406e92aec84ac585304e57bd37a40d
I20240502 21:26:52.823029 3812 raft_consensus.cc:2781] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:52.823503 3812 raft_consensus.cc:686] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Becoming Leader. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Running, Role: LEADER
I20240502 21:26:52.823526 3840 ts_tablet_manager.cc:1398] T 534b4d96df9b4bc5a9480b6c45eb11a5 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 21:26:52.823904 3842 tablet_bootstrap.cc:492] T 2068521c26dc44f9ba60e274eb039b22 P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:52.824155 3842 ts_tablet_manager.cc:1367] T 2068521c26dc44f9ba60e274eb039b22 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240502 21:26:52.824048 3812 consensus_queue.cc:237] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:52.825102 3842 raft_consensus.cc:348] T 2068521c26dc44f9ba60e274eb039b22 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:52.826198 3842 raft_consensus.cc:374] T 2068521c26dc44f9ba60e274eb039b22 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:52.826434 3842 raft_consensus.cc:729] T 2068521c26dc44f9ba60e274eb039b22 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:52.826817 31796 raft_consensus.cc:1261] T 88189733d9a14560971dbeb71543073f P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:52.826792 3842 consensus_queue.cc:260] T 2068521c26dc44f9ba60e274eb039b22 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:52.828261 3842 ts_tablet_manager.cc:1398] T 2068521c26dc44f9ba60e274eb039b22 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:26:52.840674 3769 raft_consensus.cc:482] T 534b4d96df9b4bc5a9480b6c45eb11a5 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:52.840885 3769 raft_consensus.cc:504] T 534b4d96df9b4bc5a9480b6c45eb11a5 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:52.841629 3769 leader_election.cc:288] T 534b4d96df9b4bc5a9480b6c45eb11a5 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:52.842523 31795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534b4d96df9b4bc5a9480b6c45eb11a5" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:52.843653 2095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534b4d96df9b4bc5a9480b6c45eb11a5" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:52.843914 2095 raft_consensus.cc:2379] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1406e92aec84ac585304e57bd37a40d in current term 1: Already voted for candidate 08d7a0680d244b55b93d17f9ab57b84c in this term.
I20240502 21:26:52.844377 2163 leader_election.cc:302] T 534b4d96df9b4bc5a9480b6c45eb11a5 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1406e92aec84ac585304e57bd37a40d; no voters: 08d7a0680d244b55b93d17f9ab57b84c, 1ade0354c44c4c37ae480c7a94b11e1b
I20240502 21:26:52.844970 3769 raft_consensus.cc:3037] T 534b4d96df9b4bc5a9480b6c45eb11a5 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:52.848011 3769 raft_consensus.cc:2726] T 534b4d96df9b4bc5a9480b6c45eb11a5 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:26:52.849315 3824 consensus_queue.cc:1035] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:52.854568 3824 consensus_queue.cc:1035] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:52.869917 3769 raft_consensus.cc:482] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:52.870121 3769 raft_consensus.cc:504] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:52.870934 3769 leader_election.cc:288] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:52.871400 2095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2068521c26dc44f9ba60e274eb039b22" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:52.871575 2095 raft_consensus.cc:2454] T 2068521c26dc44f9ba60e274eb039b22 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:26:52.872040 2163 leader_election.cc:302] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:52.872136 3853 tablet_replica.cc:330] T dab4f06cc78946dc962267cb2aae04f7 P d1406e92aec84ac585304e57bd37a40d: stopping tablet replica
I20240502 21:26:52.872469 3853 raft_consensus.cc:2229] T dab4f06cc78946dc962267cb2aae04f7 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:52.872541 3769 raft_consensus.cc:2781] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:52.872655 3769 raft_consensus.cc:482] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:52.872774 3769 raft_consensus.cc:3037] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:52.872778 1965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2068521c26dc44f9ba60e274eb039b22" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:26:52.873041 1965 raft_consensus.cc:2454] T 2068521c26dc44f9ba60e274eb039b22 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:26:52.877128 3769 raft_consensus.cc:504] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:52.877686 3769 leader_election.cc:288] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Requested vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:52.879570 1965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2068521c26dc44f9ba60e274eb039b22" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf"
I20240502 21:26:52.879783 1965 raft_consensus.cc:3037] T 2068521c26dc44f9ba60e274eb039b22 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:52.883229 1965 raft_consensus.cc:2454] T 2068521c26dc44f9ba60e274eb039b22 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 1.
I20240502 21:26:52.883862 2161 leader_election.cc:302] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dec645b473f40578e726f3e50e219bf, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:52.886391 3819 raft_consensus.cc:2932] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:52.887619 2095 raft_consensus.cc:1261] T acb3bdc518b84b5abc362cc4e12027c3 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:52.888516 2095 raft_consensus.cc:1261] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:52.889092 3553 consensus_queue.cc:1035] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:52.889528 31924 raft_consensus.cc:1261] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEARNER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:52.891134 1833 raft_consensus.cc:1261] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:52.893625 3403 consensus_queue.cc:1035] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:52.894238 3403 consensus_queue.cc:1035] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:52.891832 31666 raft_consensus.cc:2932] T fc93b5adc1d94e809e97e56b56822611 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:52.898120 3845 ts_tablet_manager.cc:1398] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.181s user 0.002s sys 0.000s
I20240502 21:26:52.900867 3859 raft_consensus.cc:482] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:52.901010 3859 raft_consensus.cc:504] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:52.901978 3859 leader_election.cc:288] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:26:52.903378 2094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5f7ee5a6c8a41fc80a5ef1c65a75579" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:52.903561 2094 raft_consensus.cc:2454] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:26:52.903967 31602 leader_election.cc:302] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 1ade0354c44c4c37ae480c7a94b11e1b; no voters:
I20240502 21:26:52.904088 3850 tablet_replica.cc:330] T 71974e7442b042bebc999b42833be63f P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:52.904270 3774 raft_consensus.cc:2781] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:52.904366 3774 raft_consensus.cc:482] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:52.904515 3774 raft_consensus.cc:3037] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:52.904572 1832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5f7ee5a6c8a41fc80a5ef1c65a75579" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:26:52.904778 1832 raft_consensus.cc:2454] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:26:52.905227 3850 raft_consensus.cc:2229] T 71974e7442b042bebc999b42833be63f P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:52.905786 3850 raft_consensus.cc:2258] T 71974e7442b042bebc999b42833be63f P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:52.907500 3774 raft_consensus.cc:504] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:52.908982 3816 raft_consensus.cc:1050] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c: attempting to promote NON_VOTER 1e04157b98dd4fad98858c290b73c3db to VOTER
I20240502 21:26:52.909902 3816 consensus_queue.cc:237] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } }
I20240502 21:26:52.912844 1832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5f7ee5a6c8a41fc80a5ef1c65a75579" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:26:52.913022 1832 raft_consensus.cc:3037] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:52.916154 1832 raft_consensus.cc:2454] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:52.916924 1834 raft_consensus.cc:1261] T 5b1dcae186f542cea92c9191fa8ef951 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:26:52.917673 31602 leader_election.cc:302] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 1e04157b98dd4fad98858c290b73c3db; no voters:
I20240502 21:26:52.917954 3859 raft_consensus.cc:2781] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:52.918228 3517 consensus_queue.cc:1035] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:26:52.918887 3774 leader_election.cc:288] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Requested vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:26:52.947435 2076 tablet_service.cc:1467] Processing CreateTablet for tablet 31885f3320e241018d23c5d562eb61cd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ef5a459d7998420dadc414cbac9ca38d]), partition=
I20240502 21:26:52.947818 2076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31885f3320e241018d23c5d562eb61cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:52.947878 1946 tablet_service.cc:1467] Processing CreateTablet for tablet 31885f3320e241018d23c5d562eb61cd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ef5a459d7998420dadc414cbac9ca38d]), partition=
I20240502 21:26:52.948216 1946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31885f3320e241018d23c5d562eb61cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:52.949340 3853 raft_consensus.cc:2258] T dab4f06cc78946dc962267cb2aae04f7 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:52.951983 31393 catalog_manager.cc:5484] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: term changed from 0 to 1, leader changed from <none> to 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131). New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:52.952766 31775 tablet_service.cc:1467] Processing CreateTablet for tablet 31885f3320e241018d23c5d562eb61cd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ef5a459d7998420dadc414cbac9ca38d]), partition=
I20240502 21:26:52.952620 31393 catalog_manager.cc:5484] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:52.953060 31775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31885f3320e241018d23c5d562eb61cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:52.954782 3841 tablet_bootstrap.cc:492] T 31885f3320e241018d23c5d562eb61cd P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:52.955103 3553 consensus_queue.cc:1035] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.065s
I20240502 21:26:52.956395 3841 tablet_bootstrap.cc:654] T 31885f3320e241018d23c5d562eb61cd P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
W20240502 21:26:52.958727 2163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.226.135:38569, user_credentials={real_user=slave}} blocked reactor thread for 84822.1us
I20240502 21:26:52.959496 3846 tablet_bootstrap.cc:492] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:52.960211 3842 tablet_bootstrap.cc:492] T 31885f3320e241018d23c5d562eb61cd P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:52.960995 1832 raft_consensus.cc:1261] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:52.961388 3846 tablet_bootstrap.cc:654] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:52.961660 3842 tablet_bootstrap.cc:654] T 31885f3320e241018d23c5d562eb61cd P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:52.961585 3553 consensus_queue.cc:1035] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 21:26:52.963145 2161 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.226.134:43753, user_credentials={real_user=slave}} blocked reactor thread for 79349.9us
I20240502 21:26:52.963315 2094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2068521c26dc44f9ba60e274eb039b22" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:26:52.963503 2094 raft_consensus.cc:3037] T 2068521c26dc44f9ba60e274eb039b22 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:52.963836 3765 raft_consensus.cc:2781] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:52.963865 2227 raft_consensus.cc:1261] T 5b1dcae186f542cea92c9191fa8ef951 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:26:52.963985 3765 raft_consensus.cc:686] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Becoming Leader. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Running, Role: LEADER
I20240502 21:26:52.964272 3765 consensus_queue.cc:237] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:52.964512 3517 consensus_queue.cc:1035] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:26:52.967039 2094 raft_consensus.cc:2454] T 2068521c26dc44f9ba60e274eb039b22 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 1.
I20240502 21:26:52.979496 3813 raft_consensus.cc:1050] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1: attempting to promote NON_VOTER 9dec645b473f40578e726f3e50e219bf to VOTER
I20240502 21:26:52.989897 31925 raft_consensus.cc:1261] T 5b1dcae186f542cea92c9191fa8ef951 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:26:52.993106 3853 ts_tablet_manager.cc:1875] T dab4f06cc78946dc962267cb2aae04f7 P d1406e92aec84ac585304e57bd37a40d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:52.995409 2096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5f7ee5a6c8a41fc80a5ef1c65a75579" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:26:52.995625 2096 raft_consensus.cc:3037] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:52.997910 3874 ts_tablet_manager.cc:902] T f5e82c0ab5c441efbcfbcd383b8db82f P 08d7a0680d244b55b93d17f9ab57b84c: Initiating tablet copy from peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:52.999750 2096 raft_consensus.cc:2454] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:53.005892 3517 consensus_queue.cc:1035] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:26:53.006108 3818 raft_consensus.cc:2932] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:53.007175 31535 raft_consensus.cc:1261] T acb3bdc518b84b5abc362cc4e12027c3 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEARNER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:53.009024 3504 raft_consensus.cc:2932] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:53.009124 31534 raft_consensus.cc:2932] T fc93b5adc1d94e809e97e56b56822611 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:53.013469 3207 raft_consensus.cc:2932] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:26:53.015130 31393 catalog_manager.cc:5484] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d reported cstate change: term changed from 0 to 1, leader changed from <none> to d1406e92aec84ac585304e57bd37a40d (127.24.226.136). New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:53.015856 31390 catalog_manager.cc:5484] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:53.016817 31536 raft_consensus.cc:2932] T 88189733d9a14560971dbeb71543073f P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:52.984030 3813 consensus_queue.cc:237] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:26:53.028326 3669 raft_consensus.cc:1050] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db: attempting to promote NON_VOTER 9dec645b473f40578e726f3e50e219bf to VOTER
I20240502 21:26:53.029547 3850 ts_tablet_manager.cc:1875] T 71974e7442b042bebc999b42833be63f P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:53.029395 3669 consensus_queue.cc:237] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:26:53.037468 3846 tablet_bootstrap.cc:492] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:53.038548 3846 ts_tablet_manager.cc:1367] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.079s user 0.008s sys 0.004s
I20240502 21:26:53.039283 3403 consensus_queue.cc:1035] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:53.039911 3846 raft_consensus.cc:348] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:53.043848 2095 raft_consensus.cc:2932] T 88189733d9a14560971dbeb71543073f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:53.040330 3517 raft_consensus.cc:2932] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:53.047748 31795 raft_consensus.cc:2932] T 88189733d9a14560971dbeb71543073f P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:53.050170 1832 raft_consensus.cc:2932] T 5b1dcae186f542cea92c9191fa8ef951 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:53.058025 31666 raft_consensus.cc:1261] T ae56d18c629a4e5eb50911c89de4227c P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:53.058909 3824 consensus_queue.cc:1035] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:53.063517 1965 raft_consensus.cc:1261] T ae56d18c629a4e5eb50911c89de4227c P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:53.063118 2097 raft_consensus.cc:2932] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:26:53.064550 3824 consensus_queue.cc:1035] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:53.067553 3846 raft_consensus.cc:374] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:53.067701 3846 raft_consensus.cc:729] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:53.067590 1834 raft_consensus.cc:2932] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:53.068126 3846 consensus_queue.cc:260] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
W20240502 21:26:53.072379 31471 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.226.131:40807, user_credentials={real_user=slave}} blocked reactor thread for 50797.6us
I20240502 21:26:53.073565 1833 raft_consensus.cc:2932] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:26:53.074672 3842 tablet_bootstrap.cc:492] T 31885f3320e241018d23c5d562eb61cd P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:53.074877 3842 ts_tablet_manager.cc:1367] T 31885f3320e241018d23c5d562eb61cd P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.115s user 0.020s sys 0.011s
I20240502 21:26:53.077930 3846 ts_tablet_manager.cc:1398] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.039s user 0.001s sys 0.001s
I20240502 21:26:53.078457 3841 tablet_bootstrap.cc:492] T 31885f3320e241018d23c5d562eb61cd P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:26:53.078653 3841 ts_tablet_manager.cc:1367] T 31885f3320e241018d23c5d562eb61cd P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.124s user 0.003s sys 0.009s
I20240502 21:26:53.080643 3841 raft_consensus.cc:348] T 31885f3320e241018d23c5d562eb61cd P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:53.080888 3841 raft_consensus.cc:374] T 31885f3320e241018d23c5d562eb61cd P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:53.080991 3841 raft_consensus.cc:729] T 31885f3320e241018d23c5d562eb61cd P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:26:53.081260 3841 consensus_queue.cc:260] T 31885f3320e241018d23c5d562eb61cd P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:53.081851 3841 ts_tablet_manager.cc:1398] T 31885f3320e241018d23c5d562eb61cd P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:26:53.085405 2094 raft_consensus.cc:2932] T acb3bdc518b84b5abc362cc4e12027c3 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:53.086396 31665 raft_consensus.cc:1261] T b9342dbe0497499ba4c238f2ed033293 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:53.086969 31796 raft_consensus.cc:1261] T b9342dbe0497499ba4c238f2ed033293 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:53.086946 3529 consensus_queue.cc:1035] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:53.087443 3529 consensus_queue.cc:1035] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 21:26:53.088502 31862 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.226.135:38569, user_credentials={real_user=slave}} blocked reactor thread for 45561.5us
I20240502 21:26:53.089672 31797 raft_consensus.cc:1261] T ae56d18c629a4e5eb50911c89de4227c P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:53.097462 3842 raft_consensus.cc:348] T 31885f3320e241018d23c5d562eb61cd P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:53.097671 3842 raft_consensus.cc:374] T 31885f3320e241018d23c5d562eb61cd P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:53.097751 3842 raft_consensus.cc:729] T 31885f3320e241018d23c5d562eb61cd P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:53.097986 3842 consensus_queue.cc:260] T 31885f3320e241018d23c5d562eb61cd P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:53.098493 3842 ts_tablet_manager.cc:1398] T 31885f3320e241018d23c5d562eb61cd P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.023s user 0.001s sys 0.000s
I20240502 21:26:53.099256 31924 raft_consensus.cc:2932] T 5b1dcae186f542cea92c9191fa8ef951 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:53.101167 3853 ts_tablet_manager.cc:1888] T dab4f06cc78946dc962267cb2aae04f7 P d1406e92aec84ac585304e57bd37a40d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:26:53.103018 3859 raft_consensus.cc:686] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Becoming Leader. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Running, Role: LEADER
I20240502 21:26:53.103407 1834 raft_consensus.cc:1203] T a1fd14de204c40d98027ecb329ff79f3 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:26:53.103327 3859 consensus_queue.cc:237] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:26:53.104933 3850 ts_tablet_manager.cc:1888] T 71974e7442b042bebc999b42833be63f P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:53.106676 31536 raft_consensus.cc:2932] T acb3bdc518b84b5abc362cc4e12027c3 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:53.109220 3853 log.cc:1196] T dab4f06cc78946dc962267cb2aae04f7 P d1406e92aec84ac585304e57bd37a40d: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal/wals/dab4f06cc78946dc962267cb2aae04f7
I20240502 21:26:53.109405 2230 raft_consensus.cc:1203] T 55fc18e4cc98446ca50c1dd2b6a9a89d P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:26:53.110260 31381 catalog_manager.cc:4830] TS d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): tablet dab4f06cc78946dc962267cb2aae04f7 (table rebalancer_extra_table_10 [id=112fc5f7af4e4c8080926e0a6128b468]) successfully deleted
I20240502 21:26:53.110883 3504 consensus_queue.cc:1035] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:53.111212 3853 tablet_replica.cc:330] T 71974e7442b042bebc999b42833be63f P d1406e92aec84ac585304e57bd37a40d: stopping tablet replica
I20240502 21:26:53.111385 3853 raft_consensus.cc:2229] T 71974e7442b042bebc999b42833be63f P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:53.111572 3853 raft_consensus.cc:2258] T 71974e7442b042bebc999b42833be63f P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:53.111828 3850 log.cc:1196] T 71974e7442b042bebc999b42833be63f P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/71974e7442b042bebc999b42833be63f
I20240502 21:26:53.109553 2227 raft_consensus.cc:2932] T 5b1dcae186f542cea92c9191fa8ef951 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:53.113368 3868 tablet_copy_client.cc:308] T 2db9f537acba4424bca97ffc05c6d0b4 P a44f41b0b6644fb6aa29476aed8c964f: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.132:35933
I20240502 21:26:53.114038 31947 tablet_copy_service.cc:140] P 2d50f33557944bb480ea32199d9ee4f1: Received BeginTabletCopySession request for tablet 2db9f537acba4424bca97ffc05c6d0b4 from peer a44f41b0b6644fb6aa29476aed8c964f ({username='slave'} at 127.24.226.129:33745)
I20240502 21:26:53.114231 31947 tablet_copy_service.cc:161] P 2d50f33557944bb480ea32199d9ee4f1: Beginning new tablet copy session on tablet 2db9f537acba4424bca97ffc05c6d0b4 from peer a44f41b0b6644fb6aa29476aed8c964f at {username='slave'} at 127.24.226.129:33745: session id = a44f41b0b6644fb6aa29476aed8c964f-2db9f537acba4424bca97ffc05c6d0b4
I20240502 21:26:53.114342 3850 ts_tablet_manager.cc:1909] T 71974e7442b042bebc999b42833be63f P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:26:53.115746 3853 ts_tablet_manager.cc:1875] T 71974e7442b042bebc999b42833be63f P d1406e92aec84ac585304e57bd37a40d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:53.116312 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet 71974e7442b042bebc999b42833be63f (table rebalancer_dynamic_table_new_name [id=9074dd1629f54b0e9607336d3f7224a2]) successfully deleted
I20240502 21:26:53.117533 31947 tablet_copy_source_session.cc:205] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:53.121912 3901 raft_consensus.cc:1050] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f: attempting to promote NON_VOTER 9dec645b473f40578e726f3e50e219bf to VOTER
I20240502 21:26:53.122119 31392 catalog_manager.cc:5484] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf reported cstate change: config changed from index 2 to 3, a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9dec645b473f40578e726f3e50e219bf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:53.122406 31924 raft_consensus.cc:2932] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:26:53.123442 31388 catalog_manager.cc:5484] T 5b1dcae186f542cea92c9191fa8ef951 P 1e04157b98dd4fad98858c290b73c3db reported cstate change: config changed from index 7 to 8, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:53.082231 1965 raft_consensus.cc:1261] T b9342dbe0497499ba4c238f2ed033293 P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:53.128116 3669 consensus_queue.cc:1035] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:53.128561 3813 raft_consensus.cc:2932] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:53.129217 3517 raft_consensus.cc:482] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:53.129464 3517 raft_consensus.cc:504] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:53.131062 3517 leader_election.cc:288] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:26:53.132570 1964 raft_consensus.cc:2932] T ae56d18c629a4e5eb50911c89de4227c P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:53.133699 1966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31885f3320e241018d23c5d562eb61cd" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:26:53.133921 1966 raft_consensus.cc:2454] T 31885f3320e241018d23c5d562eb61cd P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:53.133968 3868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2db9f537acba4424bca97ffc05c6d0b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:53.134282 31730 leader_election.cc:302] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:26:53.134013 31665 raft_consensus.cc:2932] T ae56d18c629a4e5eb50911c89de4227c P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:53.134634 3517 raft_consensus.cc:2781] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:53.134743 3517 raft_consensus.cc:482] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:53.134824 3517 raft_consensus.cc:3037] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:53.137341 3517 raft_consensus.cc:504] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:53.138265 2095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31885f3320e241018d23c5d562eb61cd" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:26:53.138610 2095 raft_consensus.cc:3037] T 31885f3320e241018d23c5d562eb61cd P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:53.138767 3517 leader_election.cc:288] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Requested vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:26:53.138919 1964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31885f3320e241018d23c5d562eb61cd" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf"
I20240502 21:26:53.139183 1964 raft_consensus.cc:3037] T 31885f3320e241018d23c5d562eb61cd P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:53.139114 2096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31885f3320e241018d23c5d562eb61cd" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:53.139631 3901 consensus_queue.cc:237] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:26:53.143298 31388 catalog_manager.cc:5484] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: config changed from index 4 to 5, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:53.148990 2095 raft_consensus.cc:2454] T 31885f3320e241018d23c5d562eb61cd P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:53.149411 31732 leader_election.cc:302] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 1ade0354c44c4c37ae480c7a94b11e1b; no voters:
I20240502 21:26:53.149937 3816 raft_consensus.cc:2781] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:53.150061 3816 raft_consensus.cc:686] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Becoming Leader. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Running, Role: LEADER
I20240502 21:26:53.150300 3816 consensus_queue.cc:237] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:53.151132 31387 catalog_manager.cc:5484] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: config changed from index 2 to 3, 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:26:53.154117 3819 raft_consensus.cc:1050] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c: attempting to promote NON_VOTER d1406e92aec84ac585304e57bd37a40d to VOTER
I20240502 21:26:53.154686 3853 ts_tablet_manager.cc:1888] T 71974e7442b042bebc999b42833be63f P d1406e92aec84ac585304e57bd37a40d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:53.154806 3853 log.cc:1196] T 71974e7442b042bebc999b42833be63f P d1406e92aec84ac585304e57bd37a40d: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal/wals/71974e7442b042bebc999b42833be63f
I20240502 21:26:53.154850 3819 consensus_queue.cc:237] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } }
I20240502 21:26:53.155257 3853 ts_tablet_manager.cc:1909] T 71974e7442b042bebc999b42833be63f P d1406e92aec84ac585304e57bd37a40d: Deleting consensus metadata
I20240502 21:26:53.156142 31381 catalog_manager.cc:4830] TS d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): tablet 71974e7442b042bebc999b42833be63f (table rebalancer_dynamic_table_new_name [id=9074dd1629f54b0e9607336d3f7224a2]) successfully deleted
I20240502 21:26:53.159613 3868 tablet_copy_client.cc:783] T 2db9f537acba4424bca97ffc05c6d0b4 P a44f41b0b6644fb6aa29476aed8c964f: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:53.159873 3868 tablet_copy_client.cc:647] T 2db9f537acba4424bca97ffc05c6d0b4 P a44f41b0b6644fb6aa29476aed8c964f: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:53.161650 1964 raft_consensus.cc:2454] T 31885f3320e241018d23c5d562eb61cd P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:53.162479 31536 consensus_queue.cc:237] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } }
I20240502 21:26:53.165061 2097 raft_consensus.cc:1261] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:53.165639 3911 consensus_queue.cc:1035] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:53.168560 3905 raft_consensus.cc:2932] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:26:53.172322 31382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5bc6c7211d5f4f73b2a050b9ec78c6b1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:53.175827 2095 raft_consensus.cc:2932] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:26:53.176401 31387 catalog_manager.cc:5484] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: config changed from index 3 to 4, VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) evicted. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:53.177716 31926 raft_consensus.cc:1261] T 759b7529d7d84baab1a4622d34f803bf P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:53.178267 3911 consensus_queue.cc:1035] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:53.178901 1966 raft_consensus.cc:1261] T 759b7529d7d84baab1a4622d34f803bf P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:53.180306 1813 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bc6c7211d5f4f73b2a050b9ec78c6b1 with delete_type TABLET_DATA_TOMBSTONED (TS 1e04157b98dd4fad98858c290b73c3db not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47510
I20240502 21:26:53.182188 3868 tablet_copy_client.cc:523] T 2db9f537acba4424bca97ffc05c6d0b4 P a44f41b0b6644fb6aa29476aed8c964f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:53.184346 3911 consensus_queue.cc:1035] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:53.186990 3868 tablet_bootstrap.cc:492] T 2db9f537acba4424bca97ffc05c6d0b4 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:53.188486 31797 raft_consensus.cc:1261] T 759b7529d7d84baab1a4622d34f803bf P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:53.189170 3901 consensus_queue.cc:1035] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:53.197575 31923 consensus_queue.cc:237] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } }
I20240502 21:26:53.200531 31923 consensus_queue.cc:237] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:26:53.202674 31665 raft_consensus.cc:1261] T ae56d18c629a4e5eb50911c89de4227c P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:53.203241 3813 consensus_queue.cc:1035] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:53.206149 1966 raft_consensus.cc:1261] T ae56d18c629a4e5eb50911c89de4227c P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:53.206176 3504 raft_consensus.cc:2932] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:53.206068 3911 raft_consensus.cc:2932] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:53.206735 3824 consensus_queue.cc:1035] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:53.208009 1963 raft_consensus.cc:2932] T ae56d18c629a4e5eb50911c89de4227c P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:53.210507 31665 raft_consensus.cc:2932] T ae56d18c629a4e5eb50911c89de4227c P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:53.210451 1962 raft_consensus.cc:2932] T 759b7529d7d84baab1a4622d34f803bf P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:53.214540 31394 catalog_manager.cc:5484] T ae56d18c629a4e5eb50911c89de4227c P 9dec645b473f40578e726f3e50e219bf reported cstate change: config changed from index 5 to 6, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:53.217689 31387 catalog_manager.cc:5484] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: config changed from index 4 to 5, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:53.219657 31775 tablet_service.cc:1514] Processing DeleteTablet for tablet ae56d18c629a4e5eb50911c89de4227c with delete_type TABLET_DATA_TOMBSTONED (TS 08d7a0680d244b55b93d17f9ab57b84c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51596
I20240502 21:26:53.221560 31923 raft_consensus.cc:2932] T 759b7529d7d84baab1a4622d34f803bf P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:53.230973 31536 consensus_queue.cc:237] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:26:53.233081 31382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ae56d18c629a4e5eb50911c89de4227c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:53.233163 2095 raft_consensus.cc:1261] T 88189733d9a14560971dbeb71543073f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:53.233580 1966 raft_consensus.cc:1261] T 759b7529d7d84baab1a4622d34f803bf P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:53.234278 31923 raft_consensus.cc:1261] T 759b7529d7d84baab1a4622d34f803bf P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:53.234342 3911 consensus_queue.cc:1035] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:53.234575 3504 consensus_queue.cc:1035] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:53.236642 3905 consensus_queue.cc:1035] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:53.237316 3905 raft_consensus.cc:2932] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:53.242156 1962 raft_consensus.cc:2932] T 759b7529d7d84baab1a4622d34f803bf P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:53.245700 31382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 759b7529d7d84baab1a4622d34f803bf with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:53.246796 31394 catalog_manager.cc:5484] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: config changed from index 5 to 6, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:53.248294 31926 raft_consensus.cc:2932] T 759b7529d7d84baab1a4622d34f803bf P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:53.249723 31389 catalog_manager.cc:5484] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130). New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:53.252863 31796 raft_consensus.cc:1261] T 88189733d9a14560971dbeb71543073f P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:53.253069 3517 consensus_queue.cc:1035] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:53.253266 3529 raft_consensus.cc:2932] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:53.262012 31388 catalog_manager.cc:5484] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db reported cstate change: config changed from index 5 to 6, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:53.262816 31775 tablet_service.cc:1514] Processing DeleteTablet for tablet 759b7529d7d84baab1a4622d34f803bf with delete_type TABLET_DATA_TOMBSTONED (TS 08d7a0680d244b55b93d17f9ab57b84c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51596
I20240502 21:26:53.268031 3812 raft_consensus.cc:1050] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c: attempting to promote NON_VOTER d1406e92aec84ac585304e57bd37a40d to VOTER
I20240502 21:26:53.268812 3812 consensus_queue.cc:237] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } }
I20240502 21:26:53.269438 31667 raft_consensus.cc:2932] T b9342dbe0497499ba4c238f2ed033293 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:53.272781 31796 raft_consensus.cc:2932] T b9342dbe0497499ba4c238f2ed033293 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:53.281049 3925 ts_tablet_manager.cc:902] T fc93b5adc1d94e809e97e56b56822611 P 9dec645b473f40578e726f3e50e219bf: Initiating tablet copy from peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:53.281073 31536 raft_consensus.cc:1261] T 53b7eec870c44999b48f0bf744b2ffcd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:53.281491 31388 catalog_manager.cc:5484] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: term changed from 0 to 1, leader changed from <none> to 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131). New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:53.282332 2230 raft_consensus.cc:1261] T 53b7eec870c44999b48f0bf744b2ffcd P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:53.282910 3811 consensus_queue.cc:1035] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:53.283535 3811 consensus_queue.cc:1035] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:53.284067 3504 consensus_queue.cc:1035] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:53.284288 3926 tablet_replica.cc:330] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1e04157b98dd4fad98858c290b73c3db: stopping tablet replica
I20240502 21:26:53.284623 3926 raft_consensus.cc:2229] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:53.284900 3926 raft_consensus.cc:2258] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:53.289177 31794 consensus_queue.cc:237] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } }
I20240502 21:26:53.290056 3504 raft_consensus.cc:2932] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:53.293020 1834 raft_consensus.cc:1261] T 5b1dcae186f542cea92c9191fa8ef951 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:26:53.293264 3207 raft_consensus.cc:1050] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f: attempting to promote NON_VOTER d1406e92aec84ac585304e57bd37a40d to VOTER
I20240502 21:26:53.293704 3811 consensus_queue.cc:1035] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:26:53.294162 3207 consensus_queue.cc:237] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } }
I20240502 21:26:53.295069 2097 raft_consensus.cc:2932] T 88189733d9a14560971dbeb71543073f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:53.300628 31382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 88189733d9a14560971dbeb71543073f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:53.302699 31388 catalog_manager.cc:5484] T 88189733d9a14560971dbeb71543073f P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: config changed from index 5 to 6, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:53.302718 31793 raft_consensus.cc:2932] T 88189733d9a14560971dbeb71543073f P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:53.303622 3926 ts_tablet_manager.cc:1875] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1e04157b98dd4fad98858c290b73c3db: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:53.307623 3931 tablet_replica.cc:330] T ae56d18c629a4e5eb50911c89de4227c P 08d7a0680d244b55b93d17f9ab57b84c: stopping tablet replica
I20240502 21:26:53.307875 3931 raft_consensus.cc:2229] T ae56d18c629a4e5eb50911c89de4227c P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:53.308097 3931 pending_rounds.cc:62] T ae56d18c629a4e5eb50911c89de4227c P 08d7a0680d244b55b93d17f9ab57b84c: Trying to abort 1 pending ops.
I20240502 21:26:53.308156 3931 pending_rounds.cc:68] T ae56d18c629a4e5eb50911c89de4227c P 08d7a0680d244b55b93d17f9ab57b84c: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023350632374247424 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "ae56d18c629a4e5eb50911c89de4227c" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } } }
I20240502 21:26:53.308650 3931 raft_consensus.cc:2894] T ae56d18c629a4e5eb50911c89de4227c P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 21:26:53.308781 3931 raft_consensus.cc:2866] T ae56d18c629a4e5eb50911c89de4227c P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 21:26:53.308918 3931 raft_consensus.cc:2258] T ae56d18c629a4e5eb50911c89de4227c P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:53.311367 3517 raft_consensus.cc:2932] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:53.312386 31925 raft_consensus.cc:1261] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:53.314682 3911 consensus_queue.cc:1035] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:53.318357 1833 raft_consensus.cc:2932] T 5b1dcae186f542cea92c9191fa8ef951 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:53.319540 2225 raft_consensus.cc:1261] T 5b1dcae186f542cea92c9191fa8ef951 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:26:53.321451 3926 ts_tablet_manager.cc:1888] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1e04157b98dd4fad98858c290b73c3db: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:26:53.321621 3926 log.cc:1196] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1e04157b98dd4fad98858c290b73c3db: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal/wals/5bc6c7211d5f4f73b2a050b9ec78c6b1
I20240502 21:26:53.322400 3931 ts_tablet_manager.cc:1875] T ae56d18c629a4e5eb50911c89de4227c P 08d7a0680d244b55b93d17f9ab57b84c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:53.322504 31381 catalog_manager.cc:4830] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): tablet 5bc6c7211d5f4f73b2a050b9ec78c6b1 (table rebalancer_extra_table_6 [id=86898baa76944823b3c5e8aba98d8772]) successfully deleted
I20240502 21:26:53.323096 31516 tablet_service.cc:1514] Processing DeleteTablet for tablet 88189733d9a14560971dbeb71543073f with delete_type TABLET_DATA_TOMBSTONED (TS a44f41b0b6644fb6aa29476aed8c964f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45194
I20240502 21:26:53.327528 31388 catalog_manager.cc:5484] T 5b1dcae186f542cea92c9191fa8ef951 P 1e04157b98dd4fad98858c290b73c3db reported cstate change: config changed from index 8 to 9, VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) evicted. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:53.330396 3931 ts_tablet_manager.cc:1888] T ae56d18c629a4e5eb50911c89de4227c P 08d7a0680d244b55b93d17f9ab57b84c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:26:53.330577 3931 log.cc:1196] T ae56d18c629a4e5eb50911c89de4227c P 08d7a0680d244b55b93d17f9ab57b84c: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/wals/ae56d18c629a4e5eb50911c89de4227c
I20240502 21:26:53.331431 31381 catalog_manager.cc:4830] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): tablet ae56d18c629a4e5eb50911c89de4227c (table rebalance_test_table_5 [id=9e294586148643e080e91357bcbffb17]) successfully deleted
I20240502 21:26:53.331964 3931 tablet_replica.cc:330] T 759b7529d7d84baab1a4622d34f803bf P 08d7a0680d244b55b93d17f9ab57b84c: stopping tablet replica
I20240502 21:26:53.332720 3931 raft_consensus.cc:2229] T 759b7529d7d84baab1a4622d34f803bf P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:53.333516 3931 pending_rounds.cc:62] T 759b7529d7d84baab1a4622d34f803bf P 08d7a0680d244b55b93d17f9ab57b84c: Trying to abort 1 pending ops.
I20240502 21:26:53.333882 1964 raft_consensus.cc:2932] T b9342dbe0497499ba4c238f2ed033293 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:53.337366 1833 consensus_queue.cc:237] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:26:53.341841 31666 raft_consensus.cc:1261] T 8ab82de23912483ea178376a5ba5cdc1 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:53.342427 3207 consensus_queue.cc:1035] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:53.342989 31905 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b1dcae186f542cea92c9191fa8ef951 with delete_type TABLET_DATA_TOMBSTONED (TS 2d50f33557944bb480ea32199d9ee4f1 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:47824
I20240502 21:26:53.345224 2227 raft_consensus.cc:1261] T 8ab82de23912483ea178376a5ba5cdc1 P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:53.345734 31665 raft_consensus.cc:1261] T b9342dbe0497499ba4c238f2ed033293 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:26:53.345718 3905 consensus_queue.cc:1035] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:53.346850 3931 pending_rounds.cc:68] T 759b7529d7d84baab1a4622d34f803bf P 08d7a0680d244b55b93d17f9ab57b84c: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023350632948707328 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "759b7529d7d84baab1a4622d34f803bf" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } } }
I20240502 21:26:53.347568 3931 raft_consensus.cc:2894] T 759b7529d7d84baab1a4622d34f803bf P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 21:26:53.347714 3931 raft_consensus.cc:2866] T 759b7529d7d84baab1a4622d34f803bf P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 21:26:53.347795 3669 consensus_queue.cc:1035] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:26:53.347949 3931 raft_consensus.cc:2258] T 759b7529d7d84baab1a4622d34f803bf P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:53.359289 3931 ts_tablet_manager.cc:1875] T 759b7529d7d84baab1a4622d34f803bf P 08d7a0680d244b55b93d17f9ab57b84c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:53.367408 3947 tablet_replica.cc:330] T 5b1dcae186f542cea92c9191fa8ef951 P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:26:53.368886 3947 raft_consensus.cc:2229] T 5b1dcae186f542cea92c9191fa8ef951 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:53.369182 31381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5b1dcae186f542cea92c9191fa8ef951 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:53.369380 3947 raft_consensus.cc:2258] T 5b1dcae186f542cea92c9191fa8ef951 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:53.372996 3947 ts_tablet_manager.cc:1875] T 5b1dcae186f542cea92c9191fa8ef951 P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:53.366483 31924 raft_consensus.cc:2932] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:26:53.376194 3943 tablet_replica.cc:330] T 88189733d9a14560971dbeb71543073f P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:53.377207 3943 raft_consensus.cc:2229] T 88189733d9a14560971dbeb71543073f P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:53.379415 3943 raft_consensus.cc:2258] T 88189733d9a14560971dbeb71543073f P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:53.379421 3687 raft_consensus.cc:2932] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:53.384330 31381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b9342dbe0497499ba4c238f2ed033293 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240502 21:26:53.384536 31732 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.226.133:39863, user_credentials={real_user=slave}} blocked reactor thread for 58948.5us
I20240502 21:26:53.385835 31666 raft_consensus.cc:2932] T b9342dbe0497499ba4c238f2ed033293 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:53.385262 3943 ts_tablet_manager.cc:1875] T 88189733d9a14560971dbeb71543073f P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:53.386745 31388 catalog_manager.cc:5484] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db reported cstate change: config changed from index 6 to 7, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:53.389099 31925 raft_consensus.cc:1261] T 53b7eec870c44999b48f0bf744b2ffcd P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:53.390128 3944 consensus_queue.cc:1035] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:53.390843 31775 tablet_service.cc:1514] Processing DeleteTablet for tablet b9342dbe0497499ba4c238f2ed033293 with delete_type TABLET_DATA_TOMBSTONED (TS 08d7a0680d244b55b93d17f9ab57b84c not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51596
I20240502 21:26:53.334873 3816 consensus_queue.cc:1035] T 5b1dcae186f542cea92c9191fa8ef951 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:26:53.342051 31797 raft_consensus.cc:1261] T 8ab82de23912483ea178376a5ba5cdc1 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:53.395864 1965 raft_consensus.cc:1261] T b9342dbe0497499ba4c238f2ed033293 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:26:53.399730 3669 consensus_queue.cc:1035] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:26:53.399813 2231 raft_consensus.cc:2932] T 5b1dcae186f542cea92c9191fa8ef951 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:53.401615 3947 ts_tablet_manager.cc:1888] T 5b1dcae186f542cea92c9191fa8ef951 P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:26:53.401741 3947 log.cc:1196] T 5b1dcae186f542cea92c9191fa8ef951 P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/5b1dcae186f542cea92c9191fa8ef951
I20240502 21:26:53.402900 1965 raft_consensus.cc:2932] T b9342dbe0497499ba4c238f2ed033293 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:53.404415 31388 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:26:53.406198 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet 5b1dcae186f542cea92c9191fa8ef951 (table rebalance_test_table_13 [id=c7511c25586c4ba086fcfffa353da437]) successfully deleted
I20240502 21:26:53.406389 3943 ts_tablet_manager.cc:1888] T 88189733d9a14560971dbeb71543073f P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:26:53.406517 3943 log.cc:1196] T 88189733d9a14560971dbeb71543073f P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/88189733d9a14560971dbeb71543073f
I20240502 21:26:53.407269 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet 88189733d9a14560971dbeb71543073f (table rebalance_test_table_11 [id=c46054f8169d4487b3c912c0681b1c3a]) successfully deleted
I20240502 21:26:53.411413 3765 consensus_queue.cc:1035] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:53.418449 3868 tablet_bootstrap.cc:492] T 2db9f537acba4424bca97ffc05c6d0b4 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:53.418926 3868 tablet_bootstrap.cc:492] T 2db9f537acba4424bca97ffc05c6d0b4 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap complete.
I20240502 21:26:53.435441 2097 raft_consensus.cc:1261] T 31885f3320e241018d23c5d562eb61cd P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:53.436275 3942 consensus_queue.cc:1035] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:53.439297 2231 raft_consensus.cc:1261] T 5a54d2aad63c4f3182992f00b8564acf P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:53.439816 3942 consensus_queue.cc:1035] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:53.439328 3944 raft_consensus.cc:2932] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:53.456454 3874 tablet_copy_client.cc:308] T f5e82c0ab5c441efbcfbcd383b8db82f P 08d7a0680d244b55b93d17f9ab57b84c: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.136:34907
I20240502 21:26:53.457468 2251 tablet_copy_service.cc:140] P d1406e92aec84ac585304e57bd37a40d: Received BeginTabletCopySession request for tablet f5e82c0ab5c441efbcfbcd383b8db82f from peer 08d7a0680d244b55b93d17f9ab57b84c ({username='slave'} at 127.24.226.131:53089)
I20240502 21:26:53.457691 2251 tablet_copy_service.cc:161] P d1406e92aec84ac585304e57bd37a40d: Beginning new tablet copy session on tablet f5e82c0ab5c441efbcfbcd383b8db82f from peer 08d7a0680d244b55b93d17f9ab57b84c at {username='slave'} at 127.24.226.131:53089: session id = 08d7a0680d244b55b93d17f9ab57b84c-f5e82c0ab5c441efbcfbcd383b8db82f
I20240502 21:26:53.462612 2230 raft_consensus.cc:2932] T 53b7eec870c44999b48f0bf744b2ffcd P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:53.464249 2251 tablet_copy_source_session.cc:205] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:53.458117 31536 raft_consensus.cc:2932] T 53b7eec870c44999b48f0bf744b2ffcd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:53.469195 31924 raft_consensus.cc:1261] T 5a54d2aad63c4f3182992f00b8564acf P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:53.469544 3874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5e82c0ab5c441efbcfbcd383b8db82f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:53.471417 3812 consensus_queue.cc:1035] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:53.473518 3951 consensus_queue.cc:1035] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:53.474762 3931 ts_tablet_manager.cc:1888] T 759b7529d7d84baab1a4622d34f803bf P 08d7a0680d244b55b93d17f9ab57b84c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:26:53.474946 3931 log.cc:1196] T 759b7529d7d84baab1a4622d34f803bf P 08d7a0680d244b55b93d17f9ab57b84c: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/wals/759b7529d7d84baab1a4622d34f803bf
I20240502 21:26:53.476150 31381 catalog_manager.cc:4830] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): tablet 759b7529d7d84baab1a4622d34f803bf (table rebalance_test_table_13 [id=c7511c25586c4ba086fcfffa353da437]) successfully deleted
I20240502 21:26:53.484512 3207 consensus_queue.cc:1035] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:53.485144 3207 raft_consensus.cc:2932] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:53.487974 3942 raft_consensus.cc:1050] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c: attempting to promote NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b to VOTER
I20240502 21:26:53.488922 3942 consensus_queue.cc:237] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } }
I20240502 21:26:53.490128 31667 raft_consensus.cc:2932] T 8ab82de23912483ea178376a5ba5cdc1 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:53.491937 1964 consensus_queue.cc:237] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } }
I20240502 21:26:53.493849 1966 raft_consensus.cc:1261] T 31885f3320e241018d23c5d562eb61cd P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:53.494906 31534 raft_consensus.cc:1261] T acb3bdc518b84b5abc362cc4e12027c3 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:53.495534 3902 consensus_queue.cc:1035] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:53.496160 1833 raft_consensus.cc:1261] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:53.497349 3902 consensus_queue.cc:1035] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:53.499701 3769 consensus_queue.cc:1035] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:53.500793 3818 raft_consensus.cc:2932] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:53.501621 1833 raft_consensus.cc:2932] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:53.506634 31379 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet acb3bdc518b84b5abc362cc4e12027c3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:53.507988 31534 raft_consensus.cc:2932] T acb3bdc518b84b5abc362cc4e12027c3 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:53.508144 31395 catalog_manager.cc:5484] T acb3bdc518b84b5abc362cc4e12027c3 P 9dec645b473f40578e726f3e50e219bf reported cstate change: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New cstate: current_term: 1 leader_uuid: "9dec645b473f40578e726f3e50e219bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:53.512629 2076 tablet_service.cc:1514] Processing DeleteTablet for tablet acb3bdc518b84b5abc362cc4e12027c3 with delete_type TABLET_DATA_TOMBSTONED (TS 1ade0354c44c4c37ae480c7a94b11e1b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36740
I20240502 21:26:53.498342 3811 consensus_queue.cc:1035] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.023s
I20240502 21:26:53.521817 3868 ts_tablet_manager.cc:1367] T 2db9f537acba4424bca97ffc05c6d0b4 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.335s user 0.007s sys 0.002s
I20240502 21:26:53.522166 31666 raft_consensus.cc:1261] T 5a54d2aad63c4f3182992f00b8564acf P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:53.522754 3811 consensus_queue.cc:1035] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:53.522770 3868 raft_consensus.cc:348] T 2db9f537acba4424bca97ffc05c6d0b4 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } }
I20240502 21:26:53.523046 3868 raft_consensus.cc:729] T 2db9f537acba4424bca97ffc05c6d0b4 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: LEARNER
I20240502 21:26:53.523303 3868 consensus_queue.cc:260] T 2db9f537acba4424bca97ffc05c6d0b4 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } }
I20240502 21:26:53.523941 3967 tablet_replica.cc:330] T acb3bdc518b84b5abc362cc4e12027c3 P 1ade0354c44c4c37ae480c7a94b11e1b: stopping tablet replica
I20240502 21:26:53.526306 3949 raft_consensus.cc:2932] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:53.527549 3925 tablet_copy_client.cc:308] T fc93b5adc1d94e809e97e56b56822611 P 9dec645b473f40578e726f3e50e219bf: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.131:40807
I20240502 21:26:53.528293 3931 tablet_replica.cc:330] T b9342dbe0497499ba4c238f2ed033293 P 08d7a0680d244b55b93d17f9ab57b84c: stopping tablet replica
I20240502 21:26:53.529021 31667 raft_consensus.cc:2932] T 5a54d2aad63c4f3182992f00b8564acf P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:53.530207 3931 raft_consensus.cc:2229] T b9342dbe0497499ba4c238f2ed033293 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:53.530766 31924 raft_consensus.cc:2932] T 5a54d2aad63c4f3182992f00b8564acf P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:53.530464 3967 raft_consensus.cc:2229] T acb3bdc518b84b5abc362cc4e12027c3 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:53.531879 2225 raft_consensus.cc:2932] T 8ab82de23912483ea178376a5ba5cdc1 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:53.539671 31395 catalog_manager.cc:5576] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569) does not have the latest schema for tablet 31885f3320e241018d23c5d562eb61cd (table rebalancer_dynamic_table [id=ef5a459d7998420dadc414cbac9ca38d]). Expected version 1 got 0
I20240502 21:26:53.527588 31394 catalog_manager.cc:5484] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: config changed from index 4 to 5, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:53.540992 3967 raft_consensus.cc:2258] T acb3bdc518b84b5abc362cc4e12027c3 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:53.530606 3931 raft_consensus.cc:2258] T b9342dbe0497499ba4c238f2ed033293 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:53.543696 3967 ts_tablet_manager.cc:1875] T acb3bdc518b84b5abc362cc4e12027c3 P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:53.551455 31387 catalog_manager.cc:5484] T 5a54d2aad63c4f3182992f00b8564acf P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: config changed from index 4 to 5, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:53.554054 3868 ts_tablet_manager.cc:1398] T 2db9f537acba4424bca97ffc05c6d0b4 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.032s user 0.001s sys 0.001s
I20240502 21:26:53.557296 31947 tablet_copy_service.cc:342] P 2d50f33557944bb480ea32199d9ee4f1: Request end of tablet copy session a44f41b0b6644fb6aa29476aed8c964f-2db9f537acba4424bca97ffc05c6d0b4 received from {username='slave'} at 127.24.226.129:33745
I20240502 21:26:53.557476 31947 tablet_copy_service.cc:433] P 2d50f33557944bb480ea32199d9ee4f1: ending tablet copy session a44f41b0b6644fb6aa29476aed8c964f-2db9f537acba4424bca97ffc05c6d0b4 on tablet 2db9f537acba4424bca97ffc05c6d0b4 with peer a44f41b0b6644fb6aa29476aed8c964f
I20240502 21:26:53.557737 31817 tablet_copy_service.cc:140] P 08d7a0680d244b55b93d17f9ab57b84c: Received BeginTabletCopySession request for tablet fc93b5adc1d94e809e97e56b56822611 from peer 9dec645b473f40578e726f3e50e219bf ({username='slave'} at 127.24.226.134:42809)
I20240502 21:26:53.557900 31817 tablet_copy_service.cc:161] P 08d7a0680d244b55b93d17f9ab57b84c: Beginning new tablet copy session on tablet fc93b5adc1d94e809e97e56b56822611 from peer 9dec645b473f40578e726f3e50e219bf at {username='slave'} at 127.24.226.134:42809: session id = 9dec645b473f40578e726f3e50e219bf-fc93b5adc1d94e809e97e56b56822611
I20240502 21:26:53.560592 31387 catalog_manager.cc:5576] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753) does not have the latest schema for tablet 31885f3320e241018d23c5d562eb61cd (table rebalancer_dynamic_table [id=ef5a459d7998420dadc414cbac9ca38d]). Expected version 1 got 0
I20240502 21:26:53.561743 3967 ts_tablet_manager.cc:1888] T acb3bdc518b84b5abc362cc4e12027c3 P 1ade0354c44c4c37ae480c7a94b11e1b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:26:53.561853 3967 log.cc:1196] T acb3bdc518b84b5abc362cc4e12027c3 P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal/wals/acb3bdc518b84b5abc362cc4e12027c3
I20240502 21:26:53.563426 31925 raft_consensus.cc:2932] T 53b7eec870c44999b48f0bf744b2ffcd P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:53.570691 31387 catalog_manager.cc:5484] T 53b7eec870c44999b48f0bf744b2ffcd P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: config changed from index 4 to 5, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:53.576866 2224 raft_consensus.cc:2932] T 5a54d2aad63c4f3182992f00b8564acf P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:53.588511 3975 tablet.cc:1722] T 31885f3320e241018d23c5d562eb61cd P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:53.588915 31387 catalog_manager.cc:5576] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753) does not have the latest schema for tablet 31885f3320e241018d23c5d562eb61cd (table rebalancer_dynamic_table [id=ef5a459d7998420dadc414cbac9ca38d]). Expected version 1 got 0
I20240502 21:26:53.589643 3975 tablet.cc:1620] T 31885f3320e241018d23c5d562eb61cd P 9dec645b473f40578e726f3e50e219bf: MemRowSet was empty: no flush needed.
I20240502 21:26:53.594980 3816 raft_consensus.cc:1050] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c: attempting to promote NON_VOTER 1e04157b98dd4fad98858c290b73c3db to VOTER
I20240502 21:26:53.595852 3816 consensus_queue.cc:237] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } }
I20240502 21:26:53.598062 3973 tablet.cc:1722] T 31885f3320e241018d23c5d562eb61cd P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:53.598229 31381 catalog_manager.cc:4830] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): tablet acb3bdc518b84b5abc362cc4e12027c3 (table rebalancer_extra_table_14 [id=d8ac6ebe732c415bb64d1aecfeadc621]) successfully deleted
I20240502 21:26:53.598628 3973 tablet.cc:1620] T 31885f3320e241018d23c5d562eb61cd P 1ade0354c44c4c37ae480c7a94b11e1b: MemRowSet was empty: no flush needed.
I20240502 21:26:53.580711 3931 ts_tablet_manager.cc:1875] T b9342dbe0497499ba4c238f2ed033293 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:53.600092 31389 catalog_manager.cc:5576] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569) does not have the latest schema for tablet 31885f3320e241018d23c5d562eb61cd (table rebalancer_dynamic_table [id=ef5a459d7998420dadc414cbac9ca38d]). Expected version 1 got 0
I20240502 21:26:53.604624 31817 tablet_copy_source_session.cc:205] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:53.607445 3925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc93b5adc1d94e809e97e56b56822611. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:53.607758 31394 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=ef5a459d7998420dadc414cbac9ca38d] alter complete (version 1)
I20240502 21:26:53.612172 3925 tablet_copy_client.cc:783] T fc93b5adc1d94e809e97e56b56822611 P 9dec645b473f40578e726f3e50e219bf: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:53.612398 3925 tablet_copy_client.cc:647] T fc93b5adc1d94e809e97e56b56822611 P 9dec645b473f40578e726f3e50e219bf: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:53.617434 3859 consensus_queue.cc:1035] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:53.632652 31796 raft_consensus.cc:2932] T 8ab82de23912483ea178376a5ba5cdc1 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:53.636400 31536 raft_consensus.cc:1261] T c5c344f8f84f4a73b853b4f908b93964 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:53.632251 3974 tablet.cc:1722] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:53.637653 3819 consensus_queue.cc:1035] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:53.638603 2097 raft_consensus.cc:1261] T c5c344f8f84f4a73b853b4f908b93964 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:53.639020 3974 tablet.cc:1620] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c: MemRowSet was empty: no flush needed.
I20240502 21:26:53.640105 31667 raft_consensus.cc:1261] T c5c344f8f84f4a73b853b4f908b93964 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:53.640594 3819 consensus_queue.cc:1035] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240502 21:26:53.642517 3819 consensus_queue.cc:1035] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:53.644304 3874 tablet_copy_client.cc:783] T f5e82c0ab5c441efbcfbcd383b8db82f P 08d7a0680d244b55b93d17f9ab57b84c: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:53.644726 3874 tablet_copy_client.cc:647] T f5e82c0ab5c441efbcfbcd383b8db82f P 08d7a0680d244b55b93d17f9ab57b84c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:53.652448 3874 tablet_copy_client.cc:523] T f5e82c0ab5c441efbcfbcd383b8db82f P 08d7a0680d244b55b93d17f9ab57b84c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:53.654945 3931 ts_tablet_manager.cc:1888] T b9342dbe0497499ba4c238f2ed033293 P 08d7a0680d244b55b93d17f9ab57b84c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 21:26:53.658627 31667 raft_consensus.cc:1261] T a1fd14de204c40d98027ecb329ff79f3 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:53.660072 31534 raft_consensus.cc:1261] T a1fd14de204c40d98027ecb329ff79f3 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:53.661499 1832 raft_consensus.cc:1261] T a1fd14de204c40d98027ecb329ff79f3 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:53.659217 3946 consensus_queue.cc:1035] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:53.662098 3816 consensus_queue.cc:1035] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:53.663442 3931 log.cc:1196] T b9342dbe0497499ba4c238f2ed033293 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/wals/b9342dbe0497499ba4c238f2ed033293
I20240502 21:26:53.663949 31795 consensus_queue.cc:237] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } }
I20240502 21:26:53.664917 3946 consensus_queue.cc:1035] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:53.666368 2224 raft_consensus.cc:1261] T 53b7eec870c44999b48f0bf744b2ffcd P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:53.667022 3951 consensus_queue.cc:1035] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:53.670176 31534 raft_consensus.cc:1261] T 53b7eec870c44999b48f0bf744b2ffcd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:53.670491 2224 raft_consensus.cc:2932] T 53b7eec870c44999b48f0bf744b2ffcd P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:53.677779 31389 catalog_manager.cc:5484] T 53b7eec870c44999b48f0bf744b2ffcd P d1406e92aec84ac585304e57bd37a40d reported cstate change: config changed from index 5 to 6, VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) evicted. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:53.680387 31381 catalog_manager.cc:4830] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): tablet b9342dbe0497499ba4c238f2ed033293 (table rebalancer_extra_table_0 [id=9e962597ef7c41f493f1a41cd94c913b]) successfully deleted
I20240502 21:26:53.681851 31905 tablet_service.cc:1514] Processing DeleteTablet for tablet 53b7eec870c44999b48f0bf744b2ffcd with delete_type TABLET_DATA_TOMBSTONED (TS 2d50f33557944bb480ea32199d9ee4f1 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47824
I20240502 21:26:53.682174 3947 tablet_replica.cc:330] T 53b7eec870c44999b48f0bf744b2ffcd P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:26:53.682394 3947 raft_consensus.cc:2229] T 53b7eec870c44999b48f0bf744b2ffcd P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:53.682621 3947 raft_consensus.cc:2258] T 53b7eec870c44999b48f0bf744b2ffcd P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:53.683220 3951 raft_consensus.cc:2932] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:53.683579 3947 ts_tablet_manager.cc:1875] T 53b7eec870c44999b48f0bf744b2ffcd P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:53.683882 3925 tablet_copy_client.cc:523] T fc93b5adc1d94e809e97e56b56822611 P 9dec645b473f40578e726f3e50e219bf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:53.686676 3946 consensus_queue.cc:1035] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
I20240502 21:26:53.688402 31797 consensus_queue.cc:237] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } }
I20240502 21:26:53.689059 31381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 53b7eec870c44999b48f0bf744b2ffcd with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:53.690865 31925 raft_consensus.cc:1261] T 5a54d2aad63c4f3182992f00b8564acf P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:53.691431 3946 consensus_queue.cc:1035] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:53.692080 3947 ts_tablet_manager.cc:1888] T 53b7eec870c44999b48f0bf744b2ffcd P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:26:53.692255 3947 log.cc:1196] T 53b7eec870c44999b48f0bf744b2ffcd P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/53b7eec870c44999b48f0bf744b2ffcd
I20240502 21:26:53.694725 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet 53b7eec870c44999b48f0bf744b2ffcd (table rebalance_test_table_7 [id=0af22f3605164ff58dc3914737c2de4f]) successfully deleted
I20240502 21:26:53.695152 31925 raft_consensus.cc:2932] T 5a54d2aad63c4f3182992f00b8564acf P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:53.696744 2224 raft_consensus.cc:1261] T 5a54d2aad63c4f3182992f00b8564acf P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:53.700618 31387 catalog_manager.cc:5484] T 5a54d2aad63c4f3182992f00b8564acf P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: config changed from index 5 to 6, VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) evicted. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:53.701172 3946 consensus_queue.cc:1035] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:53.695004 3942 raft_consensus.cc:2932] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:53.703804 2224 raft_consensus.cc:2932] T 5a54d2aad63c4f3182992f00b8564acf P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:53.704319 31645 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a54d2aad63c4f3182992f00b8564acf with delete_type TABLET_DATA_TOMBSTONED (TS 05f4fac41c4f4959b9b316cb112d0ada not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:53.704975 3925 tablet_bootstrap.cc:492] T fc93b5adc1d94e809e97e56b56822611 P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:53.713090 31381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5a54d2aad63c4f3182992f00b8564acf with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:53.713498 31534 raft_consensus.cc:2932] T 53b7eec870c44999b48f0bf744b2ffcd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:53.716511 3905 raft_consensus.cc:1050] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f: attempting to promote NON_VOTER d1406e92aec84ac585304e57bd37a40d to VOTER
I20240502 21:26:53.717502 3905 consensus_queue.cc:237] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } }
I20240502 21:26:53.725121 3989 tablet_replica.cc:330] T 5a54d2aad63c4f3182992f00b8564acf P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:53.726013 3989 raft_consensus.cc:2229] T 5a54d2aad63c4f3182992f00b8564acf P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:53.726297 31387 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:26:53.729950 3989 raft_consensus.cc:2258] T 5a54d2aad63c4f3182992f00b8564acf P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:53.732395 3874 tablet_bootstrap.cc:492] T f5e82c0ab5c441efbcfbcd383b8db82f P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:53.737056 3989 ts_tablet_manager.cc:1875] T 5a54d2aad63c4f3182992f00b8564acf P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:53.744990 3973 tablet.cc:1722] T 31885f3320e241018d23c5d562eb61cd P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:53.745110 3995 consensus_queue.cc:1035] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:53.745481 3973 tablet.cc:1620] T 31885f3320e241018d23c5d562eb61cd P 1ade0354c44c4c37ae480c7a94b11e1b: MemRowSet was empty: no flush needed.
I20240502 21:26:53.751175 3989 ts_tablet_manager.cc:1888] T 5a54d2aad63c4f3182992f00b8564acf P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:26:53.756879 31394 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=ef5a459d7998420dadc414cbac9ca38d] alter complete (version 2)
I20240502 21:26:53.758942 3974 tablet.cc:1722] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:53.759420 3974 tablet.cc:1620] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c: MemRowSet was empty: no flush needed.
I20240502 21:26:53.764660 3874 tablet_bootstrap.cc:492] T f5e82c0ab5c441efbcfbcd383b8db82f P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:53.765208 3874 tablet_bootstrap.cc:492] T f5e82c0ab5c441efbcfbcd383b8db82f P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap complete.
I20240502 21:26:53.765467 3874 ts_tablet_manager.cc:1367] T f5e82c0ab5c441efbcfbcd383b8db82f P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.063s user 0.008s sys 0.000s
I20240502 21:26:53.765323 3819 raft_consensus.cc:2932] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:53.770493 1833 raft_consensus.cc:2932] T a1fd14de204c40d98027ecb329ff79f3 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:53.774206 3874 raft_consensus.cc:348] T f5e82c0ab5c441efbcfbcd383b8db82f P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } }
I20240502 21:26:53.773808 31387 catalog_manager.cc:5484] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index 4 to 5, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:53.774439 3874 raft_consensus.cc:729] T f5e82c0ab5c441efbcfbcd383b8db82f P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: LEARNER
I20240502 21:26:53.774691 3874 consensus_queue.cc:260] T f5e82c0ab5c441efbcfbcd383b8db82f P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } }
I20240502 21:26:53.777251 2224 raft_consensus.cc:1261] T 55fc18e4cc98446ca50c1dd2b6a9a89d P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:26:53.777576 3975 tablet.cc:1722] T 31885f3320e241018d23c5d562eb61cd P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:53.777792 3911 consensus_queue.cc:1035] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:26:53.777940 3874 ts_tablet_manager.cc:1398] T f5e82c0ab5c441efbcfbcd383b8db82f P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20240502 21:26:53.778575 2251 tablet_copy_service.cc:342] P d1406e92aec84ac585304e57bd37a40d: Request end of tablet copy session 08d7a0680d244b55b93d17f9ab57b84c-f5e82c0ab5c441efbcfbcd383b8db82f received from {username='slave'} at 127.24.226.131:53089
I20240502 21:26:53.778743 2251 tablet_copy_service.cc:433] P d1406e92aec84ac585304e57bd37a40d: ending tablet copy session 08d7a0680d244b55b93d17f9ab57b84c-f5e82c0ab5c441efbcfbcd383b8db82f on tablet f5e82c0ab5c441efbcfbcd383b8db82f with peer 08d7a0680d244b55b93d17f9ab57b84c
I20240502 21:26:53.779318 3989 log.cc:1196] T 5a54d2aad63c4f3182992f00b8564acf P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/5a54d2aad63c4f3182992f00b8564acf
I20240502 21:26:53.780292 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet 5a54d2aad63c4f3182992f00b8564acf (table rebalance_test_table_1 [id=08ebfd17dee84efd8a264a88ead3fa46]) successfully deleted
I20240502 21:26:53.770228 31667 raft_consensus.cc:2932] T a1fd14de204c40d98027ecb329ff79f3 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:53.788149 3975 tablet.cc:1620] T 31885f3320e241018d23c5d562eb61cd P 9dec645b473f40578e726f3e50e219bf: MemRowSet was empty: no flush needed.
I20240502 21:26:53.792819 1964 raft_consensus.cc:1261] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:26:53.792982 31666 raft_consensus.cc:1261] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:26:53.793591 3207 consensus_queue.cc:1035] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:26:53.794296 3207 consensus_queue.cc:1035] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:26:53.798851 3911 raft_consensus.cc:2932] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:53.803751 2225 raft_consensus.cc:2932] T 55fc18e4cc98446ca50c1dd2b6a9a89d P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:53.811430 31534 raft_consensus.cc:2932] T a1fd14de204c40d98027ecb329ff79f3 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
W20240502 21:26:53.811707 31603 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.226.129:41039, user_credentials={real_user=slave}} blocked reactor thread for 97253.5us
I20240502 21:26:53.820634 31796 consensus_queue.cc:237] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } }
I20240502 21:26:53.823179 31387 catalog_manager.cc:5484] T 55fc18e4cc98446ca50c1dd2b6a9a89d P d1406e92aec84ac585304e57bd37a40d reported cstate change: config changed from index 7 to 8, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:53.827008 1833 raft_consensus.cc:1261] T a1fd14de204c40d98027ecb329ff79f3 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:53.827596 3946 consensus_queue.cc:1035] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:53.827669 31394 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_extra_table_22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:53.834139 3816 raft_consensus.cc:2932] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:53.836822 31533 raft_consensus.cc:1261] T a1fd14de204c40d98027ecb329ff79f3 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:53.837505 31533 raft_consensus.cc:1203] T 2db9f537acba4424bca97ffc05c6d0b4 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:26:53.838434 3942 consensus_queue.cc:1035] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:53.838738 1964 raft_consensus.cc:2932] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:53.840330 2095 raft_consensus.cc:2932] T c5c344f8f84f4a73b853b4f908b93964 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:53.844566 3942 raft_consensus.cc:2932] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:53.848817 31533 raft_consensus.cc:2932] T a1fd14de204c40d98027ecb329ff79f3 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:53.858959 31381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a1fd14de204c40d98027ecb329ff79f3 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:53.862116 31395 catalog_manager.cc:5484] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index 5 to 6, VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) evicted. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:53.862501 31667 raft_consensus.cc:2932] T c5c344f8f84f4a73b853b4f908b93964 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:53.871582 31392 catalog_manager.cc:5484] T c5c344f8f84f4a73b853b4f908b93964 P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:53.874645 1834 raft_consensus.cc:2932] T a1fd14de204c40d98027ecb329ff79f3 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:53.876030 31645 tablet_service.cc:1514] Processing DeleteTablet for tablet a1fd14de204c40d98027ecb329ff79f3 with delete_type TABLET_DATA_TOMBSTONED (TS 05f4fac41c4f4959b9b316cb112d0ada not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:53.876330 3989 tablet_replica.cc:330] T a1fd14de204c40d98027ecb329ff79f3 P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:53.876538 3989 raft_consensus.cc:2229] T a1fd14de204c40d98027ecb329ff79f3 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:53.876732 3989 raft_consensus.cc:2258] T a1fd14de204c40d98027ecb329ff79f3 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:53.877681 3989 ts_tablet_manager.cc:1875] T a1fd14de204c40d98027ecb329ff79f3 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:53.882871 31536 raft_consensus.cc:2932] T c5c344f8f84f4a73b853b4f908b93964 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:53.883852 3989 ts_tablet_manager.cc:1888] T a1fd14de204c40d98027ecb329ff79f3 P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:26:53.884039 3989 log.cc:1196] T a1fd14de204c40d98027ecb329ff79f3 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/a1fd14de204c40d98027ecb329ff79f3
I20240502 21:26:53.885900 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet a1fd14de204c40d98027ecb329ff79f3 (table rebalance_test_table_5 [id=9e294586148643e080e91357bcbffb17]) successfully deleted
I20240502 21:26:53.888463 31533 consensus_queue.cc:237] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } }
I20240502 21:26:53.892082 2225 raft_consensus.cc:1261] T 55fc18e4cc98446ca50c1dd2b6a9a89d P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:26:53.892629 2631 consensus_queue.cc:1035] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:26:53.892674 1964 raft_consensus.cc:1261] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:26:53.893807 31395 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:26:53.896533 3911 consensus_queue.cc:1035] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:26:53.899994 2631 raft_consensus.cc:2932] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:53.901136 31796 consensus_queue.cc:237] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } }
I20240502 21:26:53.907016 2230 raft_consensus.cc:2932] T 55fc18e4cc98446ca50c1dd2b6a9a89d P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:53.914754 2094 raft_consensus.cc:1261] T c5c344f8f84f4a73b853b4f908b93964 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:53.915544 3951 consensus_queue.cc:1035] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:53.915475 1964 raft_consensus.cc:2932] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:53.919041 2076 tablet_service.cc:1467] Processing CreateTablet for tablet 4bc968ae4dcc4636a49257b6d5824bee (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=27be64652cc14b9fa97100370e65428b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:53.919468 2076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bc968ae4dcc4636a49257b6d5824bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:53.921198 3819 raft_consensus.cc:2932] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:53.921772 2094 raft_consensus.cc:2932] T c5c344f8f84f4a73b853b4f908b93964 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:53.922919 1813 tablet_service.cc:1467] Processing CreateTablet for tablet 84ba90ce8dd24ea2b325b1dba2e4c494 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=27be64652cc14b9fa97100370e65428b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:53.923283 1813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84ba90ce8dd24ea2b325b1dba2e4c494. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:53.925171 31516 tablet_service.cc:1467] Processing CreateTablet for tablet 4bc968ae4dcc4636a49257b6d5824bee (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=27be64652cc14b9fa97100370e65428b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:53.925547 31516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bc968ae4dcc4636a49257b6d5824bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:53.927624 31393 catalog_manager.cc:5484] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 9dec645b473f40578e726f3e50e219bf reported cstate change: config changed from index 8 to 9, VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) evicted. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:53.928608 31381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c5c344f8f84f4a73b853b4f908b93964 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:53.928647 2208 tablet_service.cc:1467] Processing CreateTablet for tablet 84ba90ce8dd24ea2b325b1dba2e4c494 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=27be64652cc14b9fa97100370e65428b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:53.929114 2208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84ba90ce8dd24ea2b325b1dba2e4c494. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:53.929674 1946 tablet_service.cc:1467] Processing CreateTablet for tablet 4bc968ae4dcc4636a49257b6d5824bee (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=27be64652cc14b9fa97100370e65428b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:53.929736 1945 tablet_service.cc:1467] Processing CreateTablet for tablet 84ba90ce8dd24ea2b325b1dba2e4c494 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=27be64652cc14b9fa97100370e65428b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:53.930073 1946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bc968ae4dcc4636a49257b6d5824bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:53.931283 1945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84ba90ce8dd24ea2b325b1dba2e4c494. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:53.933049 2210 tablet_service.cc:1467] Processing CreateTablet for tablet 1f743c7345184feaa50871dc49a1f136 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=27be64652cc14b9fa97100370e65428b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:53.933413 2210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f743c7345184feaa50871dc49a1f136. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:53.935688 31666 raft_consensus.cc:1261] T c5c344f8f84f4a73b853b4f908b93964 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:53.936295 3946 consensus_queue.cc:1035] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:53.937754 31666 raft_consensus.cc:2932] T c5c344f8f84f4a73b853b4f908b93964 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:53.939272 31515 tablet_service.cc:1467] Processing CreateTablet for tablet 1f743c7345184feaa50871dc49a1f136 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=27be64652cc14b9fa97100370e65428b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:53.939658 31515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f743c7345184feaa50871dc49a1f136. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:53.939489 31395 catalog_manager.cc:5484] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index 5 to 6, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:53.943480 31382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 55fc18e4cc98446ca50c1dd2b6a9a89d with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:53.945223 3974 tablet.cc:1722] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:53.947356 3973 tablet.cc:1722] T 31885f3320e241018d23c5d562eb61cd P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:53.950790 31514 tablet_service.cc:1514] Processing DeleteTablet for tablet c5c344f8f84f4a73b853b4f908b93964 with delete_type TABLET_DATA_TOMBSTONED (TS a44f41b0b6644fb6aa29476aed8c964f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45194
I20240502 21:26:53.956912 3975 tablet.cc:1722] T 31885f3320e241018d23c5d562eb61cd P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:53.963017 31645 tablet_service.cc:1514] Processing DeleteTablet for tablet 55fc18e4cc98446ca50c1dd2b6a9a89d with delete_type TABLET_DATA_TOMBSTONED (TS 05f4fac41c4f4959b9b316cb112d0ada not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:53.963378 3989 tablet_replica.cc:330] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:53.963574 3989 raft_consensus.cc:2229] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:53.963737 3989 pending_rounds.cc:62] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 05f4fac41c4f4959b9b316cb112d0ada: Trying to abort 1 pending ops.
I20240502 21:26:53.963929 3989 pending_rounds.cc:68] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 05f4fac41c4f4959b9b316cb112d0ada: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7023350635384508416 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "55fc18e4cc98446ca50c1dd2b6a9a89d" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } } }
I20240502 21:26:53.964406 3989 raft_consensus.cc:2894] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240502 21:26:53.964552 3989 raft_consensus.cc:2866] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 21:26:53.964684 3989 raft_consensus.cc:2258] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:53.965801 3989 ts_tablet_manager.cc:1875] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:53.971699 3989 ts_tablet_manager.cc:1888] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:26:53.971861 3989 log.cc:1196] T 55fc18e4cc98446ca50c1dd2b6a9a89d P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/55fc18e4cc98446ca50c1dd2b6a9a89d
I20240502 21:26:53.972677 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet 55fc18e4cc98446ca50c1dd2b6a9a89d (table rebalance_test_table_11 [id=c46054f8169d4487b3c912c0681b1c3a]) successfully deleted
I20240502 21:26:53.977797 4004 tablet_bootstrap.cc:492] T 84ba90ce8dd24ea2b325b1dba2e4c494 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:53.978317 4007 tablet_replica.cc:330] T c5c344f8f84f4a73b853b4f908b93964 P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:53.978825 4010 tablet_bootstrap.cc:492] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:53.980265 31386 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table_new_name [id=ef5a459d7998420dadc414cbac9ca38d] alter complete (version 3)
I20240502 21:26:53.982308 31905 tablet_service.cc:1467] Processing CreateTablet for tablet 1f743c7345184feaa50871dc49a1f136 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=27be64652cc14b9fa97100370e65428b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:53.982560 4008 tablet_bootstrap.cc:492] T 84ba90ce8dd24ea2b325b1dba2e4c494 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:53.982722 31905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f743c7345184feaa50871dc49a1f136. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:53.984640 4009 tablet_bootstrap.cc:492] T 4bc968ae4dcc4636a49257b6d5824bee P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:53.979282 4007 raft_consensus.cc:2229] T c5c344f8f84f4a73b853b4f908b93964 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:53.988396 4007 raft_consensus.cc:2258] T c5c344f8f84f4a73b853b4f908b93964 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:53.988767 4006 tablet_bootstrap.cc:492] T 4bc968ae4dcc4636a49257b6d5824bee P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:53.989679 4008 tablet_bootstrap.cc:654] T 84ba90ce8dd24ea2b325b1dba2e4c494 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:53.991187 4009 tablet_bootstrap.cc:654] T 4bc968ae4dcc4636a49257b6d5824bee P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:53.991616 4007 ts_tablet_manager.cc:1875] T c5c344f8f84f4a73b853b4f908b93964 P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:53.993062 4004 tablet_bootstrap.cc:654] T 84ba90ce8dd24ea2b325b1dba2e4c494 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:53.998760 4006 tablet_bootstrap.cc:654] T 4bc968ae4dcc4636a49257b6d5824bee P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:53.999789 4010 tablet_bootstrap.cc:654] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:54.009910 4008 tablet_bootstrap.cc:492] T 84ba90ce8dd24ea2b325b1dba2e4c494 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:54.010627 4009 tablet_bootstrap.cc:492] T 4bc968ae4dcc4636a49257b6d5824bee P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:54.010819 4008 ts_tablet_manager.cc:1367] T 84ba90ce8dd24ea2b325b1dba2e4c494 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.004s
I20240502 21:26:54.011009 4009 ts_tablet_manager.cc:1367] T 4bc968ae4dcc4636a49257b6d5824bee P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.000s
I20240502 21:26:54.012609 4008 raft_consensus.cc:348] T 84ba90ce8dd24ea2b325b1dba2e4c494 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:54.012344 4009 raft_consensus.cc:348] T 4bc968ae4dcc4636a49257b6d5824bee P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:54.012980 4008 raft_consensus.cc:374] T 84ba90ce8dd24ea2b325b1dba2e4c494 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:54.013108 4009 raft_consensus.cc:374] T 4bc968ae4dcc4636a49257b6d5824bee P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:54.013252 4008 raft_consensus.cc:729] T 84ba90ce8dd24ea2b325b1dba2e4c494 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:54.013386 4009 raft_consensus.cc:729] T 4bc968ae4dcc4636a49257b6d5824bee P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:54.013744 4008 consensus_queue.cc:260] T 84ba90ce8dd24ea2b325b1dba2e4c494 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:54.014133 4009 consensus_queue.cc:260] T 4bc968ae4dcc4636a49257b6d5824bee P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:54.014791 4008 ts_tablet_manager.cc:1398] T 84ba90ce8dd24ea2b325b1dba2e4c494 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240502 21:26:54.015115 4007 ts_tablet_manager.cc:1888] T c5c344f8f84f4a73b853b4f908b93964 P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:26:54.016053 4009 ts_tablet_manager.cc:1398] T 4bc968ae4dcc4636a49257b6d5824bee P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 21:26:54.020678 4006 tablet_bootstrap.cc:492] T 4bc968ae4dcc4636a49257b6d5824bee P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:26:54.021481 4015 tablet_bootstrap.cc:492] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:54.023737 4007 log.cc:1196] T c5c344f8f84f4a73b853b4f908b93964 P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/c5c344f8f84f4a73b853b4f908b93964
I20240502 21:26:54.025064 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet c5c344f8f84f4a73b853b4f908b93964 (table rebalance_test_table_13 [id=c7511c25586c4ba086fcfffa353da437]) successfully deleted
I20240502 21:26:54.025652 4006 ts_tablet_manager.cc:1367] T 4bc968ae4dcc4636a49257b6d5824bee P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.037s user 0.005s sys 0.000s
I20240502 21:26:54.026074 4010 tablet_bootstrap.cc:492] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:54.026276 4010 ts_tablet_manager.cc:1367] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.005s
I20240502 21:26:54.026610 4006 raft_consensus.cc:348] T 4bc968ae4dcc4636a49257b6d5824bee P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:54.027086 4010 raft_consensus.cc:348] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:54.027354 4010 raft_consensus.cc:374] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:54.027487 4010 raft_consensus.cc:729] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:54.027755 4010 consensus_queue.cc:260] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:54.032509 4010 ts_tablet_manager.cc:1398] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 21:26:54.032509 4006 raft_consensus.cc:374] T 4bc968ae4dcc4636a49257b6d5824bee P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:54.032614 4006 raft_consensus.cc:729] T 4bc968ae4dcc4636a49257b6d5824bee P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:26:54.032893 4006 consensus_queue.cc:260] T 4bc968ae4dcc4636a49257b6d5824bee P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:54.033160 4015 tablet_bootstrap.cc:654] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:54.036207 4010 tablet_bootstrap.cc:492] T 1f743c7345184feaa50871dc49a1f136 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:54.038024 4010 tablet_bootstrap.cc:654] T 1f743c7345184feaa50871dc49a1f136 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:54.038378 4006 ts_tablet_manager.cc:1398] T 4bc968ae4dcc4636a49257b6d5824bee P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240502 21:26:54.038890 3529 raft_consensus.cc:482] T 84ba90ce8dd24ea2b325b1dba2e4c494 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:54.039053 3529 raft_consensus.cc:504] T 84ba90ce8dd24ea2b325b1dba2e4c494 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:54.039764 3529 leader_election.cc:288] T 84ba90ce8dd24ea2b325b1dba2e4c494 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:54.040174 4004 tablet_bootstrap.cc:492] T 84ba90ce8dd24ea2b325b1dba2e4c494 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:54.040572 1964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ba90ce8dd24ea2b325b1dba2e4c494" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:26:54.041092 2225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ba90ce8dd24ea2b325b1dba2e4c494" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
W20240502 21:26:54.041529 1767 leader_election.cc:341] T 84ba90ce8dd24ea2b325b1dba2e4c494 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:54.041728 4006 tablet_bootstrap.cc:492] T 84ba90ce8dd24ea2b325b1dba2e4c494 P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
W20240502 21:26:54.043615 1769 leader_election.cc:341] T 84ba90ce8dd24ea2b325b1dba2e4c494 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:54.043764 1769 leader_election.cc:302] T 84ba90ce8dd24ea2b325b1dba2e4c494 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db; no voters: 9dec645b473f40578e726f3e50e219bf, d1406e92aec84ac585304e57bd37a40d
I20240502 21:26:54.044054 3529 raft_consensus.cc:2726] T 84ba90ce8dd24ea2b325b1dba2e4c494 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:26:54.040421 4004 ts_tablet_manager.cc:1367] T 84ba90ce8dd24ea2b325b1dba2e4c494 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.063s user 0.002s sys 0.007s
I20240502 21:26:54.046098 4004 raft_consensus.cc:348] T 84ba90ce8dd24ea2b325b1dba2e4c494 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:54.046343 4004 raft_consensus.cc:374] T 84ba90ce8dd24ea2b325b1dba2e4c494 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:54.046468 4004 raft_consensus.cc:729] T 84ba90ce8dd24ea2b325b1dba2e4c494 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:54.046743 4004 consensus_queue.cc:260] T 84ba90ce8dd24ea2b325b1dba2e4c494 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:54.048127 4004 ts_tablet_manager.cc:1398] T 84ba90ce8dd24ea2b325b1dba2e4c494 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:26:54.048411 4004 tablet_bootstrap.cc:492] T 1f743c7345184feaa50871dc49a1f136 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:54.049036 4006 tablet_bootstrap.cc:654] T 84ba90ce8dd24ea2b325b1dba2e4c494 P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:54.051354 4004 tablet_bootstrap.cc:654] T 1f743c7345184feaa50871dc49a1f136 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:54.053359 4015 tablet_bootstrap.cc:492] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:54.053592 4015 ts_tablet_manager.cc:1367] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20240502 21:26:54.054682 4015 raft_consensus.cc:348] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:54.055452 4015 raft_consensus.cc:374] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:54.055608 4015 raft_consensus.cc:729] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:54.055913 4015 consensus_queue.cc:260] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:54.058403 4015 ts_tablet_manager.cc:1398] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 21:26:54.061928 4004 tablet_bootstrap.cc:492] T 1f743c7345184feaa50871dc49a1f136 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:54.062345 4004 ts_tablet_manager.cc:1367] T 1f743c7345184feaa50871dc49a1f136 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.005s
I20240502 21:26:54.063211 4004 raft_consensus.cc:348] T 1f743c7345184feaa50871dc49a1f136 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:54.064255 4010 tablet_bootstrap.cc:492] T 1f743c7345184feaa50871dc49a1f136 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:54.064515 4010 ts_tablet_manager.cc:1367] T 1f743c7345184feaa50871dc49a1f136 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.001s
I20240502 21:26:54.066918 4010 raft_consensus.cc:348] T 1f743c7345184feaa50871dc49a1f136 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:54.067206 4010 raft_consensus.cc:374] T 1f743c7345184feaa50871dc49a1f136 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:54.067335 4010 raft_consensus.cc:729] T 1f743c7345184feaa50871dc49a1f136 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:54.067593 4010 consensus_queue.cc:260] T 1f743c7345184feaa50871dc49a1f136 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:54.068176 4010 ts_tablet_manager.cc:1398] T 1f743c7345184feaa50871dc49a1f136 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 21:26:54.068400 4004 raft_consensus.cc:374] T 1f743c7345184feaa50871dc49a1f136 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:54.068534 4004 raft_consensus.cc:729] T 1f743c7345184feaa50871dc49a1f136 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:54.068835 4004 consensus_queue.cc:260] T 1f743c7345184feaa50871dc49a1f136 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:54.069655 3925 tablet_bootstrap.cc:492] T fc93b5adc1d94e809e97e56b56822611 P 9dec645b473f40578e726f3e50e219bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:54.072346 31536 consensus_queue.cc:237] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } }
I20240502 21:26:54.073366 3925 tablet_bootstrap.cc:492] T fc93b5adc1d94e809e97e56b56822611 P 9dec645b473f40578e726f3e50e219bf: Bootstrap complete.
I20240502 21:26:54.074189 4006 tablet_bootstrap.cc:492] T 84ba90ce8dd24ea2b325b1dba2e4c494 P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:26:54.074412 4006 ts_tablet_manager.cc:1367] T 84ba90ce8dd24ea2b325b1dba2e4c494 P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.000s
I20240502 21:26:54.075187 2225 raft_consensus.cc:1261] T 8ab82de23912483ea178376a5ba5cdc1 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:54.075453 31797 raft_consensus.cc:1261] T 8ab82de23912483ea178376a5ba5cdc1 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:54.075709 3911 consensus_queue.cc:1035] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:54.076239 4006 raft_consensus.cc:348] T 84ba90ce8dd24ea2b325b1dba2e4c494 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:54.076527 4006 raft_consensus.cc:374] T 84ba90ce8dd24ea2b325b1dba2e4c494 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:54.076833 3911 consensus_queue.cc:1035] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:54.078037 3911 raft_consensus.cc:2932] T 8ab82de23912483ea178376a5ba5cdc1 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:54.080698 4004 ts_tablet_manager.cc:1398] T 1f743c7345184feaa50871dc49a1f136 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240502 21:26:54.082298 31388 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:54.082521 31388 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:54.084453 2225 raft_consensus.cc:2932] T 8ab82de23912483ea178376a5ba5cdc1 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:54.084952 31797 raft_consensus.cc:2932] T 8ab82de23912483ea178376a5ba5cdc1 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:54.089263 31388 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 31885f3320e241018d23c5d562eb61cd
I20240502 21:26:54.090505 31775 tablet_service.cc:1514] Processing DeleteTablet for tablet 31885f3320e241018d23c5d562eb61cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:54 UTC) from {username='slave'} at 127.0.0.1:51596
I20240502 21:26:54.090574 2076 tablet_service.cc:1514] Processing DeleteTablet for tablet 31885f3320e241018d23c5d562eb61cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:54 UTC) from {username='slave'} at 127.0.0.1:36740
I20240502 21:26:54.090757 3931 tablet_replica.cc:330] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c: stopping tablet replica
I20240502 21:26:54.090834 3967 tablet_replica.cc:330] T 31885f3320e241018d23c5d562eb61cd P 1ade0354c44c4c37ae480c7a94b11e1b: stopping tablet replica
I20240502 21:26:54.090950 3931 raft_consensus.cc:2229] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:54.091028 3967 raft_consensus.cc:2229] T 31885f3320e241018d23c5d562eb61cd P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:54.091240 3931 raft_consensus.cc:2258] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:54.091277 3967 raft_consensus.cc:2258] T 31885f3320e241018d23c5d562eb61cd P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:54.092211 3967 ts_tablet_manager.cc:1875] T 31885f3320e241018d23c5d562eb61cd P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:54.093848 3931 ts_tablet_manager.cc:1875] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:54.096745 31395 catalog_manager.cc:5484] T 8ab82de23912483ea178376a5ba5cdc1 P d1406e92aec84ac585304e57bd37a40d reported cstate change: config changed from index 5 to 6, VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) evicted. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:54.098812 31382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8ab82de23912483ea178376a5ba5cdc1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:54.100517 31645 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ab82de23912483ea178376a5ba5cdc1 with delete_type TABLET_DATA_TOMBSTONED (TS 05f4fac41c4f4959b9b316cb112d0ada not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:54.100893 3989 tablet_replica.cc:330] T 8ab82de23912483ea178376a5ba5cdc1 P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:54.101081 3989 raft_consensus.cc:2229] T 8ab82de23912483ea178376a5ba5cdc1 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:54.101296 3989 raft_consensus.cc:2258] T 8ab82de23912483ea178376a5ba5cdc1 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:54.101608 3967 ts_tablet_manager.cc:1888] T 31885f3320e241018d23c5d562eb61cd P 1ade0354c44c4c37ae480c7a94b11e1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:54.101703 3967 log.cc:1196] T 31885f3320e241018d23c5d562eb61cd P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal/wals/31885f3320e241018d23c5d562eb61cd
I20240502 21:26:54.102140 3967 ts_tablet_manager.cc:1909] T 31885f3320e241018d23c5d562eb61cd P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting consensus metadata
I20240502 21:26:54.092936 1946 tablet_service.cc:1514] Processing DeleteTablet for tablet 31885f3320e241018d23c5d562eb61cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:54 UTC) from {username='slave'} at 127.0.0.1:33002
I20240502 21:26:54.103304 3989 ts_tablet_manager.cc:1875] T 8ab82de23912483ea178376a5ba5cdc1 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:54.103413 3931 ts_tablet_manager.cc:1888] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:54.103508 3931 log.cc:1196] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/wals/31885f3320e241018d23c5d562eb61cd
I20240502 21:26:54.103947 3931 ts_tablet_manager.cc:1909] T 31885f3320e241018d23c5d562eb61cd P 08d7a0680d244b55b93d17f9ab57b84c: Deleting consensus metadata
I20240502 21:26:54.104588 4006 raft_consensus.cc:729] T 84ba90ce8dd24ea2b325b1dba2e4c494 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:26:54.104957 31381 catalog_manager.cc:4830] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): tablet 31885f3320e241018d23c5d562eb61cd (table rebalancer_dynamic_table_new_name [id=ef5a459d7998420dadc414cbac9ca38d]) successfully deleted
I20240502 21:26:54.104944 4006 consensus_queue.cc:260] T 84ba90ce8dd24ea2b325b1dba2e4c494 P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:54.105737 4006 ts_tablet_manager.cc:1398] T 84ba90ce8dd24ea2b325b1dba2e4c494 P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.031s user 0.001s sys 0.000s
I20240502 21:26:54.107025 31381 catalog_manager.cc:4830] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): tablet 31885f3320e241018d23c5d562eb61cd (table rebalancer_dynamic_table_new_name [id=ef5a459d7998420dadc414cbac9ca38d]) successfully deleted
I20240502 21:26:54.107313 4024 tablet_replica.cc:330] T 31885f3320e241018d23c5d562eb61cd P 9dec645b473f40578e726f3e50e219bf: stopping tablet replica
I20240502 21:26:54.107517 4024 raft_consensus.cc:2229] T 31885f3320e241018d23c5d562eb61cd P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:54.107764 4024 raft_consensus.cc:2258] T 31885f3320e241018d23c5d562eb61cd P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:54.074174 3925 ts_tablet_manager.cc:1367] T fc93b5adc1d94e809e97e56b56822611 P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.369s user 0.050s sys 0.004s
I20240502 21:26:54.108951 3925 raft_consensus.cc:348] T fc93b5adc1d94e809e97e56b56822611 P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:26:54.109218 3925 raft_consensus.cc:729] T fc93b5adc1d94e809e97e56b56822611 P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: LEARNER
I20240502 21:26:54.109457 3925 consensus_queue.cc:260] T fc93b5adc1d94e809e97e56b56822611 P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:26:54.110051 3925 ts_tablet_manager.cc:1398] T fc93b5adc1d94e809e97e56b56822611 P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:54.110270 4024 ts_tablet_manager.cc:1875] T 31885f3320e241018d23c5d562eb61cd P 9dec645b473f40578e726f3e50e219bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:54.113198 3989 ts_tablet_manager.cc:1888] T 8ab82de23912483ea178376a5ba5cdc1 P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:26:54.113358 3989 log.cc:1196] T 8ab82de23912483ea178376a5ba5cdc1 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/8ab82de23912483ea178376a5ba5cdc1
I20240502 21:26:54.114130 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet 8ab82de23912483ea178376a5ba5cdc1 (table rebalance_test_table_15 [id=7ff12e5a4c674a5fb7e07cb593390fd1]) successfully deleted
I20240502 21:26:54.118619 31817 tablet_copy_service.cc:342] P 08d7a0680d244b55b93d17f9ab57b84c: Request end of tablet copy session 9dec645b473f40578e726f3e50e219bf-fc93b5adc1d94e809e97e56b56822611 received from {username='slave'} at 127.24.226.134:42809
I20240502 21:26:54.118736 31817 tablet_copy_service.cc:433] P 08d7a0680d244b55b93d17f9ab57b84c: ending tablet copy session 9dec645b473f40578e726f3e50e219bf-fc93b5adc1d94e809e97e56b56822611 on tablet fc93b5adc1d94e809e97e56b56822611 with peer 9dec645b473f40578e726f3e50e219bf
I20240502 21:26:54.126251 4024 ts_tablet_manager.cc:1888] T 31885f3320e241018d23c5d562eb61cd P 9dec645b473f40578e726f3e50e219bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:54.126355 4024 log.cc:1196] T 31885f3320e241018d23c5d562eb61cd P 9dec645b473f40578e726f3e50e219bf: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal/wals/31885f3320e241018d23c5d562eb61cd
I20240502 21:26:54.126704 4024 ts_tablet_manager.cc:1909] T 31885f3320e241018d23c5d562eb61cd P 9dec645b473f40578e726f3e50e219bf: Deleting consensus metadata
I20240502 21:26:54.127467 31379 catalog_manager.cc:4830] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): tablet 31885f3320e241018d23c5d562eb61cd (table rebalancer_dynamic_table_new_name [id=ef5a459d7998420dadc414cbac9ca38d]) successfully deleted
I20240502 21:26:54.144593 1964 raft_consensus.cc:1203] T fc93b5adc1d94e809e97e56b56822611 P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:26:54.148588 2891 raft_consensus.cc:482] T 84ba90ce8dd24ea2b325b1dba2e4c494 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:54.148730 2891 raft_consensus.cc:504] T 84ba90ce8dd24ea2b325b1dba2e4c494 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:54.149423 2891 leader_election.cc:288] T 84ba90ce8dd24ea2b325b1dba2e4c494 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:26:54.149788 1964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ba90ce8dd24ea2b325b1dba2e4c494" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:26:54.149991 1964 raft_consensus.cc:2454] T 84ba90ce8dd24ea2b325b1dba2e4c494 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:26:54.150095 1834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ba90ce8dd24ea2b325b1dba2e4c494" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:26:54.150295 1834 raft_consensus.cc:2454] T 84ba90ce8dd24ea2b325b1dba2e4c494 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:26:54.150390 2161 leader_election.cc:302] T 84ba90ce8dd24ea2b325b1dba2e4c494 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dec645b473f40578e726f3e50e219bf, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:54.150677 2891 raft_consensus.cc:2781] T 84ba90ce8dd24ea2b325b1dba2e4c494 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:54.150769 2891 raft_consensus.cc:482] T 84ba90ce8dd24ea2b325b1dba2e4c494 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:54.150832 2891 raft_consensus.cc:3037] T 84ba90ce8dd24ea2b325b1dba2e4c494 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:54.152763 3824 raft_consensus.cc:482] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:54.152907 3824 raft_consensus.cc:504] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:54.153604 3824 leader_election.cc:288] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:54.154244 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f743c7345184feaa50871dc49a1f136" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:54.154475 31536 raft_consensus.cc:2454] T 1f743c7345184feaa50871dc49a1f136 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:54.155007 2891 raft_consensus.cc:504] T 84ba90ce8dd24ea2b325b1dba2e4c494 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:54.155125 31863 leader_election.cc:302] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:54.156000 1964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ba90ce8dd24ea2b325b1dba2e4c494" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf"
I20240502 21:26:54.156347 1964 raft_consensus.cc:3037] T 84ba90ce8dd24ea2b325b1dba2e4c494 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:54.156764 2225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f743c7345184feaa50871dc49a1f136" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:26:54.157034 2225 raft_consensus.cc:2454] T 1f743c7345184feaa50871dc49a1f136 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:54.157647 3504 raft_consensus.cc:2781] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:54.157729 3504 raft_consensus.cc:482] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:54.157812 3504 raft_consensus.cc:3037] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:54.159547 1964 raft_consensus.cc:2454] T 84ba90ce8dd24ea2b325b1dba2e4c494 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 1.
I20240502 21:26:54.160449 2161 leader_election.cc:302] T 84ba90ce8dd24ea2b325b1dba2e4c494 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dec645b473f40578e726f3e50e219bf, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:54.160727 3769 raft_consensus.cc:2781] T 84ba90ce8dd24ea2b325b1dba2e4c494 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:54.160872 3769 raft_consensus.cc:686] T 84ba90ce8dd24ea2b325b1dba2e4c494 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Becoming Leader. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Running, Role: LEADER
I20240502 21:26:54.160851 1834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ba90ce8dd24ea2b325b1dba2e4c494" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:26:54.161048 1834 raft_consensus.cc:3037] T 84ba90ce8dd24ea2b325b1dba2e4c494 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:54.161119 3769 consensus_queue.cc:237] T 84ba90ce8dd24ea2b325b1dba2e4c494 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:54.162824 2891 leader_election.cc:288] T 84ba90ce8dd24ea2b325b1dba2e4c494 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Requested vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:26:54.163786 1834 raft_consensus.cc:2454] T 84ba90ce8dd24ea2b325b1dba2e4c494 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 1.
I20240502 21:26:54.167323 3504 raft_consensus.cc:504] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:54.168462 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f743c7345184feaa50871dc49a1f136" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:26:54.168681 31536 raft_consensus.cc:3037] T 1f743c7345184feaa50871dc49a1f136 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:54.171679 31536 raft_consensus.cc:2454] T 1f743c7345184feaa50871dc49a1f136 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:26:54.171926 3504 leader_election.cc:288] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Requested vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:54.172487 2225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f743c7345184feaa50871dc49a1f136" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:26:54.172698 2225 raft_consensus.cc:3037] T 1f743c7345184feaa50871dc49a1f136 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:54.175546 2225 raft_consensus.cc:2454] T 1f743c7345184feaa50871dc49a1f136 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:26:54.176034 31862 leader_election.cc:302] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:54.176129 31387 catalog_manager.cc:5484] T 84ba90ce8dd24ea2b325b1dba2e4c494 P d1406e92aec84ac585304e57bd37a40d reported cstate change: term changed from 0 to 1, leader changed from <none> to d1406e92aec84ac585304e57bd37a40d (127.24.226.136). New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:54.176407 3504 raft_consensus.cc:2781] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:54.176699 3504 raft_consensus.cc:686] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Running, Role: LEADER
I20240502 21:26:54.177346 3504 consensus_queue.cc:237] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:54.190840 31387 catalog_manager.cc:5484] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132). New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:54.197649 31387 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:26:54.204728 1946 tablet_service.cc:1467] Processing CreateTablet for tablet 1d98617b2344473d83eb448adee33dcd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ed617dcec8e44a93a1d030202f77c0b6]), partition=
I20240502 21:26:54.204742 1813 tablet_service.cc:1467] Processing CreateTablet for tablet 1d98617b2344473d83eb448adee33dcd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ed617dcec8e44a93a1d030202f77c0b6]), partition=
I20240502 21:26:54.205139 1946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d98617b2344473d83eb448adee33dcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:54.205154 1813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d98617b2344473d83eb448adee33dcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:54.208734 2076 tablet_service.cc:1467] Processing CreateTablet for tablet 1d98617b2344473d83eb448adee33dcd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ed617dcec8e44a93a1d030202f77c0b6]), partition=
I20240502 21:26:54.209064 2076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d98617b2344473d83eb448adee33dcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:54.209905 4006 tablet_bootstrap.cc:492] T 1d98617b2344473d83eb448adee33dcd P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:54.211320 4006 tablet_bootstrap.cc:654] T 1d98617b2344473d83eb448adee33dcd P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:54.216377 4009 tablet_bootstrap.cc:492] T 1d98617b2344473d83eb448adee33dcd P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:54.218019 4009 tablet_bootstrap.cc:654] T 1d98617b2344473d83eb448adee33dcd P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:54.220530 4009 tablet_bootstrap.cc:492] T 1d98617b2344473d83eb448adee33dcd P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:54.220659 4009 ts_tablet_manager.cc:1367] T 1d98617b2344473d83eb448adee33dcd P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:26:54.220654 4008 tablet_bootstrap.cc:492] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:54.221652 4009 raft_consensus.cc:348] T 1d98617b2344473d83eb448adee33dcd P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:54.221868 4009 raft_consensus.cc:374] T 1d98617b2344473d83eb448adee33dcd P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:54.221928 4009 raft_consensus.cc:729] T 1d98617b2344473d83eb448adee33dcd P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:54.222560 4008 tablet_bootstrap.cc:654] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:54.224005 4009 consensus_queue.cc:260] T 1d98617b2344473d83eb448adee33dcd P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:54.224692 4009 ts_tablet_manager.cc:1398] T 1d98617b2344473d83eb448adee33dcd P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 21:26:54.228996 4008 tablet_bootstrap.cc:492] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:54.229167 4008 ts_tablet_manager.cc:1367] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240502 21:26:54.229928 4008 raft_consensus.cc:348] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:54.230227 4008 raft_consensus.cc:374] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:54.230324 4008 raft_consensus.cc:729] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:54.230544 4008 consensus_queue.cc:260] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:54.231000 4008 ts_tablet_manager.cc:1398] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:54.231503 4006 tablet_bootstrap.cc:492] T 1d98617b2344473d83eb448adee33dcd P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:26:54.231676 4006 ts_tablet_manager.cc:1367] T 1d98617b2344473d83eb448adee33dcd P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.022s user 0.018s sys 0.000s
I20240502 21:26:54.232403 4006 raft_consensus.cc:348] T 1d98617b2344473d83eb448adee33dcd P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:54.232605 4006 raft_consensus.cc:374] T 1d98617b2344473d83eb448adee33dcd P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:54.232718 4006 raft_consensus.cc:729] T 1d98617b2344473d83eb448adee33dcd P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:26:54.232931 4006 consensus_queue.cc:260] T 1d98617b2344473d83eb448adee33dcd P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:54.233490 4006 ts_tablet_manager.cc:1398] T 1d98617b2344473d83eb448adee33dcd P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:54.269940 3529 raft_consensus.cc:482] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:54.270119 3529 raft_consensus.cc:504] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:54.271869 3818 raft_consensus.cc:482] T 1d98617b2344473d83eb448adee33dcd P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:54.272001 3818 raft_consensus.cc:504] T 1d98617b2344473d83eb448adee33dcd P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:54.272348 3529 leader_election.cc:288] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:54.272545 3818 leader_election.cc:288] T 1d98617b2344473d83eb448adee33dcd P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:54.272660 1964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d98617b2344473d83eb448adee33dcd" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:26:54.272853 1964 raft_consensus.cc:2454] T 1d98617b2344473d83eb448adee33dcd P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 0.
I20240502 21:26:54.273131 2097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d98617b2344473d83eb448adee33dcd" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:54.273204 1767 leader_election.cc:302] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:26:54.273360 2097 raft_consensus.cc:2454] T 1d98617b2344473d83eb448adee33dcd P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 0.
I20240502 21:26:54.273473 3529 raft_consensus.cc:2781] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:54.273576 3529 raft_consensus.cc:482] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:54.273680 3529 raft_consensus.cc:3037] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:54.273766 1901 leader_election.cc:302] T 1d98617b2344473d83eb448adee33dcd P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:26:54.273756 1834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d98617b2344473d83eb448adee33dcd" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:26:54.274153 2094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d98617b2344473d83eb448adee33dcd" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:54.274334 2094 raft_consensus.cc:2454] T 1d98617b2344473d83eb448adee33dcd P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 0.
I20240502 21:26:54.276014 3529 raft_consensus.cc:504] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:54.276358 1834 raft_consensus.cc:2379] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9dec645b473f40578e726f3e50e219bf in current term 1: Already voted for candidate 1e04157b98dd4fad98858c290b73c3db in this term.
I20240502 21:26:54.276973 1964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d98617b2344473d83eb448adee33dcd" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf"
I20240502 21:26:54.277115 3529 leader_election.cc:288] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Requested vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:54.277153 1964 raft_consensus.cc:3037] T 1d98617b2344473d83eb448adee33dcd P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:54.277602 2094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d98617b2344473d83eb448adee33dcd" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:26:54.277760 2094 raft_consensus.cc:3037] T 1d98617b2344473d83eb448adee33dcd P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:54.279584 1964 raft_consensus.cc:2454] T 1d98617b2344473d83eb448adee33dcd P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:26:54.280129 2094 raft_consensus.cc:2454] T 1d98617b2344473d83eb448adee33dcd P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:26:54.280577 1767 leader_election.cc:302] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:26:54.280894 3818 raft_consensus.cc:2741] T 1d98617b2344473d83eb448adee33dcd P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240502 21:26:54.281162 3529 raft_consensus.cc:2781] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:54.281394 3529 raft_consensus.cc:686] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Becoming Leader. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Running, Role: LEADER
I20240502 21:26:54.281672 3529 consensus_queue.cc:237] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:54.285962 3911 raft_consensus.cc:482] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:54.286075 3911 raft_consensus.cc:504] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:54.286612 3911 leader_election.cc:288] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:26:54.286916 2094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bc968ae4dcc4636a49257b6d5824bee" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:54.287087 1964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bc968ae4dcc4636a49257b6d5824bee" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:26:54.287163 2094 raft_consensus.cc:2454] T 4bc968ae4dcc4636a49257b6d5824bee P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:54.287278 1964 raft_consensus.cc:2454] T 4bc968ae4dcc4636a49257b6d5824bee P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:54.287768 31471 leader_election.cc:302] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:54.288054 3911 raft_consensus.cc:2781] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:54.288177 3911 raft_consensus.cc:482] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:54.288267 3911 raft_consensus.cc:3037] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:54.288059 31388 catalog_manager.cc:5484] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e04157b98dd4fad98858c290b73c3db (127.24.226.133). New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:54.290686 3911 raft_consensus.cc:504] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:54.292680 2094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bc968ae4dcc4636a49257b6d5824bee" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:26:54.292770 3911 leader_election.cc:288] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Requested vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:26:54.292893 2094 raft_consensus.cc:3037] T 4bc968ae4dcc4636a49257b6d5824bee P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:54.293181 1964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bc968ae4dcc4636a49257b6d5824bee" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf"
I20240502 21:26:54.293351 1964 raft_consensus.cc:3037] T 4bc968ae4dcc4636a49257b6d5824bee P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:54.295403 1964 raft_consensus.cc:2454] T 4bc968ae4dcc4636a49257b6d5824bee P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:54.295761 31469 leader_election.cc:302] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dec645b473f40578e726f3e50e219bf, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:54.296000 3911 raft_consensus.cc:2781] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:54.296310 2094 raft_consensus.cc:2454] T 4bc968ae4dcc4636a49257b6d5824bee P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:54.296564 3911 raft_consensus.cc:686] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:26:54.297000 3911 consensus_queue.cc:237] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:54.300509 31388 catalog_manager.cc:5484] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 0 to 1, leader changed from <none> to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:54.302551 3504 raft_consensus.cc:1050] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1: attempting to promote NON_VOTER a44f41b0b6644fb6aa29476aed8c964f to VOTER
I20240502 21:26:54.303369 3504 consensus_queue.cc:237] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } }
I20240502 21:26:54.313774 2094 raft_consensus.cc:1261] T 2db9f537acba4424bca97ffc05c6d0b4 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:54.314164 1834 raft_consensus.cc:1261] T 2db9f537acba4424bca97ffc05c6d0b4 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:54.314908 3824 consensus_queue.cc:1035] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:54.315322 31533 raft_consensus.cc:1261] T 2db9f537acba4424bca97ffc05c6d0b4 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEARNER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:54.315476 3824 consensus_queue.cc:1035] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:54.315907 3813 consensus_queue.cc:1035] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:54.319906 4078 raft_consensus.cc:2932] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:54.320654 31536 raft_consensus.cc:2932] T 2db9f537acba4424bca97ffc05c6d0b4 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:54.328665 1833 raft_consensus.cc:2932] T 2db9f537acba4424bca97ffc05c6d0b4 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:54.329044 31387 catalog_manager.cc:5484] T 2db9f537acba4424bca97ffc05c6d0b4 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: config changed from index 2 to 3, a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:54.332516 2094 raft_consensus.cc:2932] T 2db9f537acba4424bca97ffc05c6d0b4 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:54.339731 31925 consensus_queue.cc:237] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } }
I20240502 21:26:54.341197 2094 raft_consensus.cc:1261] T 2db9f537acba4424bca97ffc05c6d0b4 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:54.341394 31536 raft_consensus.cc:1261] T 2db9f537acba4424bca97ffc05c6d0b4 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:54.341645 4078 consensus_queue.cc:1035] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:54.341898 3824 consensus_queue.cc:1035] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:54.344684 3824 raft_consensus.cc:2932] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:54.344753 2094 raft_consensus.cc:2932] T 2db9f537acba4424bca97ffc05c6d0b4 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:54.348090 31382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2db9f537acba4424bca97ffc05c6d0b4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:54.350533 31388 catalog_manager.cc:5484] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: config changed from index 3 to 4, VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) evicted. New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:54.353103 31536 raft_consensus.cc:2932] T 2db9f537acba4424bca97ffc05c6d0b4 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:54.356032 1813 tablet_service.cc:1514] Processing DeleteTablet for tablet 2db9f537acba4424bca97ffc05c6d0b4 with delete_type TABLET_DATA_TOMBSTONED (TS 1e04157b98dd4fad98858c290b73c3db not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47510
I20240502 21:26:54.358978 31797 raft_consensus.cc:1203] T f5e82c0ab5c441efbcfbcd383b8db82f P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:26:54.370316 4085 tablet_replica.cc:330] T 2db9f537acba4424bca97ffc05c6d0b4 P 1e04157b98dd4fad98858c290b73c3db: stopping tablet replica
I20240502 21:26:54.370527 4085 raft_consensus.cc:2229] T 2db9f537acba4424bca97ffc05c6d0b4 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:54.370848 4085 raft_consensus.cc:2258] T 2db9f537acba4424bca97ffc05c6d0b4 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:54.371711 4085 ts_tablet_manager.cc:1875] T 2db9f537acba4424bca97ffc05c6d0b4 P 1e04157b98dd4fad98858c290b73c3db: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:54.378038 4085 ts_tablet_manager.cc:1888] T 2db9f537acba4424bca97ffc05c6d0b4 P 1e04157b98dd4fad98858c290b73c3db: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:26:54.378209 4085 log.cc:1196] T 2db9f537acba4424bca97ffc05c6d0b4 P 1e04157b98dd4fad98858c290b73c3db: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal/wals/2db9f537acba4424bca97ffc05c6d0b4
I20240502 21:26:54.378990 31381 catalog_manager.cc:4830] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): tablet 2db9f537acba4424bca97ffc05c6d0b4 (table rebalancer_extra_table_4 [id=5fea5cf0a15d489c9c9a523859a65476]) successfully deleted
I20240502 21:26:54.406838 31388 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:26:54.419167 1965 raft_consensus.cc:1261] T 1d98617b2344473d83eb448adee33dcd P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:54.420099 3669 consensus_queue.cc:1035] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:54.420325 2094 raft_consensus.cc:1261] T 1d98617b2344473d83eb448adee33dcd P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:54.421464 3669 consensus_queue.cc:1035] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:54.425480 3975 tablet.cc:1722] T 1d98617b2344473d83eb448adee33dcd P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:54.425935 3975 tablet.cc:1620] T 1d98617b2344473d83eb448adee33dcd P 9dec645b473f40578e726f3e50e219bf: MemRowSet was empty: no flush needed.
I20240502 21:26:54.426261 31387 catalog_manager.cc:5576] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753) does not have the latest schema for tablet 1d98617b2344473d83eb448adee33dcd (table rebalancer_dynamic_table [id=ed617dcec8e44a93a1d030202f77c0b6]). Expected version 1 got 0
I20240502 21:26:54.428951 31387 catalog_manager.cc:5576] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569) does not have the latest schema for tablet 1d98617b2344473d83eb448adee33dcd (table rebalancer_dynamic_table [id=ed617dcec8e44a93a1d030202f77c0b6]). Expected version 1 got 0
I20240502 21:26:54.429383 3973 tablet.cc:1722] T 1d98617b2344473d83eb448adee33dcd P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:54.429505 4117 tablet.cc:1722] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:54.429812 3973 tablet.cc:1620] T 1d98617b2344473d83eb448adee33dcd P 1ade0354c44c4c37ae480c7a94b11e1b: MemRowSet was empty: no flush needed.
I20240502 21:26:54.430123 4117 tablet.cc:1620] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db: MemRowSet was empty: no flush needed.
I20240502 21:26:54.436694 31388 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=ed617dcec8e44a93a1d030202f77c0b6] alter complete (version 1)
I20240502 21:26:54.455129 2094 consensus_queue.cc:237] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } }
I20240502 21:26:54.457264 1965 raft_consensus.cc:1261] T 46279270b2334633906af57c4b43e969 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer 1ade0354c44c4c37ae480c7a94b11e1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:54.457484 2225 raft_consensus.cc:1261] T 46279270b2334633906af57c4b43e969 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 1ade0354c44c4c37ae480c7a94b11e1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:26:54.457659 2033 consensus_peers.cc:485] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): Couldn't send request to peer 05f4fac41c4f4959b9b316cb112d0ada. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 46279270b2334633906af57c4b43e969. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:54.458022 4120 consensus_queue.cc:1035] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:54.458263 3866 consensus_queue.cc:1035] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:54.460750 4120 raft_consensus.cc:2932] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } } }
I20240502 21:26:54.461231 1965 raft_consensus.cc:2932] T 46279270b2334633906af57c4b43e969 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } } }
I20240502 21:26:54.461616 2225 raft_consensus.cc:2932] T 46279270b2334633906af57c4b43e969 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } } }
I20240502 21:26:54.467018 31388 catalog_manager.cc:5484] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New cstate: current_term: 1 leader_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:54.489195 31666 consensus_queue.cc:237] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:26:54.491291 31797 raft_consensus.cc:1261] T 0b8d119ee8474a40ac08df61df5b8423 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:54.491340 31925 raft_consensus.cc:1261] T 0b8d119ee8474a40ac08df61df5b8423 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:26:54.491706 31600 consensus_peers.cc:485] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0b8d119ee8474a40ac08df61df5b8423. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:54.491740 3859 consensus_queue.cc:1035] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:54.491981 3993 consensus_queue.cc:1035] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:54.494558 3993 raft_consensus.cc:2932] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:54.495208 31925 raft_consensus.cc:2932] T 0b8d119ee8474a40ac08df61df5b8423 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:54.495436 31797 raft_consensus.cc:2932] T 0b8d119ee8474a40ac08df61df5b8423 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:54.500569 31388 catalog_manager.cc:5484] T 0b8d119ee8474a40ac08df61df5b8423 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:54.520310 31536 consensus_queue.cc:237] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } }
I20240502 21:26:54.522327 31925 raft_consensus.cc:1261] T b20df8896a2b423cb53c2c1a2a1b98e2 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:26:54.522372 31471 consensus_peers.cc:485] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f -> Peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): Couldn't send request to peer d1406e92aec84ac585304e57bd37a40d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b20df8896a2b423cb53c2c1a2a1b98e2. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:54.522388 31797 raft_consensus.cc:1261] T b20df8896a2b423cb53c2c1a2a1b98e2 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:54.523017 3207 consensus_queue.cc:1035] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:54.523511 3207 consensus_queue.cc:1035] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:54.525657 3207 raft_consensus.cc:2932] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } } }
I20240502 21:26:54.526749 31797 raft_consensus.cc:2932] T b20df8896a2b423cb53c2c1a2a1b98e2 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } } }
I20240502 21:26:54.530365 31395 catalog_manager.cc:5484] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: config changed from index -1 to 4, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:54.526515 31925 raft_consensus.cc:2932] T b20df8896a2b423cb53c2c1a2a1b98e2 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } } }
I20240502 21:26:54.543148 31395 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:26:54.549995 4117 tablet.cc:1722] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:54.550428 4117 tablet.cc:1620] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db: MemRowSet was empty: no flush needed.
I20240502 21:26:54.550539 3973 tablet.cc:1722] T 1d98617b2344473d83eb448adee33dcd P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:54.551019 3973 tablet.cc:1620] T 1d98617b2344473d83eb448adee33dcd P 1ade0354c44c4c37ae480c7a94b11e1b: MemRowSet was empty: no flush needed.
I20240502 21:26:54.554368 3975 tablet.cc:1722] T 1d98617b2344473d83eb448adee33dcd P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:54.554873 3975 tablet.cc:1620] T 1d98617b2344473d83eb448adee33dcd P 9dec645b473f40578e726f3e50e219bf: MemRowSet was empty: no flush needed.
I20240502 21:26:54.555601 2225 consensus_queue.cc:237] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } }
W20240502 21:26:54.558183 2164 consensus_peers.cc:485] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d -> Peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): Couldn't send request to peer 05f4fac41c4f4959b9b316cb112d0ada. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1022c4c6ee8642ca95250933a940fe7a. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:54.559947 2097 raft_consensus.cc:1261] T 1022c4c6ee8642ca95250933a940fe7a P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:54.560457 2891 consensus_queue.cc:1035] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:54.560982 1965 raft_consensus.cc:1261] T 1022c4c6ee8642ca95250933a940fe7a P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:54.561920 31393 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=ed617dcec8e44a93a1d030202f77c0b6] alter complete (version 2)
I20240502 21:26:54.563587 2891 consensus_queue.cc:1035] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:54.564249 2891 raft_consensus.cc:2932] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } } }
I20240502 21:26:54.564988 2097 raft_consensus.cc:2932] T 1022c4c6ee8642ca95250933a940fe7a P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } } }
I20240502 21:26:54.570331 31388 catalog_manager.cc:5484] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d reported cstate change: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:54.574823 1965 raft_consensus.cc:2932] T 1022c4c6ee8642ca95250933a940fe7a P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } } }
I20240502 21:26:54.581418 3769 consensus_queue.cc:1035] T 84ba90ce8dd24ea2b325b1dba2e4c494 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:54.591320 31667 consensus_queue.cc:237] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } }
I20240502 21:26:54.592653 2891 consensus_queue.cc:1035] T 84ba90ce8dd24ea2b325b1dba2e4c494 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:54.602993 2097 raft_consensus.cc:1261] T de2216a74f8344779a881651b66a85e5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:54.602977 1833 raft_consensus.cc:1261] T de2216a74f8344779a881651b66a85e5 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:26:54.603361 31602 consensus_peers.cc:485] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): Couldn't send request to peer 08d7a0680d244b55b93d17f9ab57b84c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de2216a74f8344779a881651b66a85e5. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:54.603688 4134 consensus_queue.cc:1035] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:54.604022 3774 consensus_queue.cc:1035] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:54.606284 4078 consensus_queue.cc:1035] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:54.606817 4134 raft_consensus.cc:2932] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } } }
I20240502 21:26:54.607583 1833 raft_consensus.cc:2932] T de2216a74f8344779a881651b66a85e5 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } } }
I20240502 21:26:54.615324 31393 catalog_manager.cc:5484] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: config changed from index -1 to 2, NON_VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) added. New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:54.617228 3951 raft_consensus.cc:1050] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c: attempting to promote NON_VOTER 9dec645b473f40578e726f3e50e219bf to VOTER
I20240502 21:26:54.618013 3951 consensus_queue.cc:237] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:26:54.624708 31667 raft_consensus.cc:1261] T fc93b5adc1d94e809e97e56b56822611 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:54.624872 31536 raft_consensus.cc:1261] T fc93b5adc1d94e809e97e56b56822611 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:54.625936 3946 consensus_queue.cc:1035] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:54.626261 4078 consensus_queue.cc:1035] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:54.626449 3946 consensus_queue.cc:1035] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:54.627139 1965 raft_consensus.cc:1261] T fc93b5adc1d94e809e97e56b56822611 P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:54.627839 3819 consensus_queue.cc:1035] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:54.647604 3946 raft_consensus.cc:2932] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:54.648386 31533 raft_consensus.cc:2932] T fc93b5adc1d94e809e97e56b56822611 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:54.656906 31393 catalog_manager.cc:5484] T fc93b5adc1d94e809e97e56b56822611 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: config changed from index 4 to 5, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:54.668368 1965 raft_consensus.cc:2932] T fc93b5adc1d94e809e97e56b56822611 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:54.669754 31667 raft_consensus.cc:2932] T fc93b5adc1d94e809e97e56b56822611 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:54.673745 31796 consensus_queue.cc:237] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:26:54.675688 31387 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:26:54.676007 2097 raft_consensus.cc:2932] T de2216a74f8344779a881651b66a85e5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } } }
I20240502 21:26:54.679562 31667 raft_consensus.cc:1261] T fc93b5adc1d94e809e97e56b56822611 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:54.687295 4117 tablet.cc:1722] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:54.687978 3975 tablet.cc:1722] T 1d98617b2344473d83eb448adee33dcd P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:54.688560 3973 tablet.cc:1722] T 1d98617b2344473d83eb448adee33dcd P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:54.692595 3946 consensus_queue.cc:1035] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:54.693447 1965 raft_consensus.cc:1261] T fc93b5adc1d94e809e97e56b56822611 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:54.695240 3819 consensus_queue.cc:1035] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:54.696027 3946 raft_consensus.cc:2932] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:54.696976 31666 raft_consensus.cc:2932] T fc93b5adc1d94e809e97e56b56822611 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:54.700157 31387 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table_new_name [id=ed617dcec8e44a93a1d030202f77c0b6] alter complete (version 3)
I20240502 21:26:54.703967 31381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fc93b5adc1d94e809e97e56b56822611 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:54.706009 31393 catalog_manager.cc:5484] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index 5 to 6, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:54.707834 31666 consensus_queue.cc:237] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:26:54.707979 1964 raft_consensus.cc:2932] T fc93b5adc1d94e809e97e56b56822611 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:54.710026 1833 raft_consensus.cc:1261] T 182b3f15c06c4c07ade0a8854d3ecafd P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 21:26:54.710390 31600 consensus_peers.cc:485] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 182b3f15c06c4c07ade0a8854d3ecafd. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:54.710553 31533 raft_consensus.cc:1261] T 182b3f15c06c4c07ade0a8854d3ecafd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:26:54.711228 3993 consensus_queue.cc:1035] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:26:54.711494 4134 consensus_queue.cc:1035] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:26:54.713060 31516 tablet_service.cc:1514] Processing DeleteTablet for tablet fc93b5adc1d94e809e97e56b56822611 with delete_type TABLET_DATA_TOMBSTONED (TS a44f41b0b6644fb6aa29476aed8c964f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45194
I20240502 21:26:54.723451 4179 tablet_replica.cc:330] T fc93b5adc1d94e809e97e56b56822611 P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:54.724628 4179 raft_consensus.cc:2229] T fc93b5adc1d94e809e97e56b56822611 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:54.725008 4179 raft_consensus.cc:2258] T fc93b5adc1d94e809e97e56b56822611 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:54.725991 4179 ts_tablet_manager.cc:1875] T fc93b5adc1d94e809e97e56b56822611 P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:54.726866 3993 raft_consensus.cc:2932] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:54.728252 1834 raft_consensus.cc:2932] T 182b3f15c06c4c07ade0a8854d3ecafd P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:54.731276 31536 raft_consensus.cc:2932] T 182b3f15c06c4c07ade0a8854d3ecafd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:54.733628 4179 ts_tablet_manager.cc:1888] T fc93b5adc1d94e809e97e56b56822611 P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:26:54.733799 4179 log.cc:1196] T fc93b5adc1d94e809e97e56b56822611 P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/fc93b5adc1d94e809e97e56b56822611
I20240502 21:26:54.734668 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet fc93b5adc1d94e809e97e56b56822611 (table rebalance_test_table_9 [id=5eee55eb24e94c988b4ddbfe2f8442d3]) successfully deleted
I20240502 21:26:54.734974 31395 catalog_manager.cc:5484] T 182b3f15c06c4c07ade0a8854d3ecafd P 1e04157b98dd4fad98858c290b73c3db reported cstate change: config changed from index 6 to 7, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:54.754052 3911 consensus_queue.cc:1035] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:54.761106 31536 consensus_queue.cc:237] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } }
I20240502 21:26:54.761668 3207 consensus_queue.cc:1035] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:54.764205 31925 raft_consensus.cc:1261] T 1ed225bdefc64ddd8eef2109bfd969df P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:54.765012 3207 consensus_queue.cc:1035] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:54.765197 31796 raft_consensus.cc:1261] T 1ed225bdefc64ddd8eef2109bfd969df P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:26:54.765384 31471 consensus_peers.cc:485] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f -> Peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): Couldn't send request to peer d1406e92aec84ac585304e57bd37a40d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ed225bdefc64ddd8eef2109bfd969df. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:54.765837 3207 consensus_queue.cc:1035] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:54.780790 4188 raft_consensus.cc:2932] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } } }
I20240502 21:26:54.781474 31796 raft_consensus.cc:2932] T 1ed225bdefc64ddd8eef2109bfd969df P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } } }
I20240502 21:26:54.783844 31923 raft_consensus.cc:2932] T 1ed225bdefc64ddd8eef2109bfd969df P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } } }
I20240502 21:26:54.789271 31387 catalog_manager.cc:5484] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: config changed from index -1 to 4, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:54.803684 31393 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:54.803864 31393 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:54.808167 31393 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 1d98617b2344473d83eb448adee33dcd
I20240502 21:26:54.808807 1813 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d98617b2344473d83eb448adee33dcd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:54 UTC) from {username='slave'} at 127.0.0.1:47510
I20240502 21:26:54.809139 4085 tablet_replica.cc:330] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db: stopping tablet replica
I20240502 21:26:54.809413 4085 raft_consensus.cc:2229] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:54.809880 4085 raft_consensus.cc:2258] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:54.810794 1946 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d98617b2344473d83eb448adee33dcd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:54 UTC) from {username='slave'} at 127.0.0.1:33002
I20240502 21:26:54.810971 4085 ts_tablet_manager.cc:1875] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:54.811967 2076 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d98617b2344473d83eb448adee33dcd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:54 UTC) from {username='slave'} at 127.0.0.1:36740
I20240502 21:26:54.820147 4085 ts_tablet_manager.cc:1888] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:54.820592 4085 log.cc:1196] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal/wals/1d98617b2344473d83eb448adee33dcd
I20240502 21:26:54.821251 4085 ts_tablet_manager.cc:1909] T 1d98617b2344473d83eb448adee33dcd P 1e04157b98dd4fad98858c290b73c3db: Deleting consensus metadata
I20240502 21:26:54.823916 31381 catalog_manager.cc:4830] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): tablet 1d98617b2344473d83eb448adee33dcd (table rebalancer_dynamic_table_new_name [id=ed617dcec8e44a93a1d030202f77c0b6]) successfully deleted
I20240502 21:26:54.824412 4199 tablet_replica.cc:330] T 1d98617b2344473d83eb448adee33dcd P 1ade0354c44c4c37ae480c7a94b11e1b: stopping tablet replica
I20240502 21:26:54.824792 4199 raft_consensus.cc:2229] T 1d98617b2344473d83eb448adee33dcd P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:54.825047 4199 raft_consensus.cc:2258] T 1d98617b2344473d83eb448adee33dcd P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:54.826005 4199 ts_tablet_manager.cc:1875] T 1d98617b2344473d83eb448adee33dcd P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:54.828429 4198 tablet_replica.cc:330] T 1d98617b2344473d83eb448adee33dcd P 9dec645b473f40578e726f3e50e219bf: stopping tablet replica
I20240502 21:26:54.829329 4198 raft_consensus.cc:2229] T 1d98617b2344473d83eb448adee33dcd P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:54.829910 4198 raft_consensus.cc:2258] T 1d98617b2344473d83eb448adee33dcd P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:54.831055 4198 ts_tablet_manager.cc:1875] T 1d98617b2344473d83eb448adee33dcd P 9dec645b473f40578e726f3e50e219bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:54.832535 4199 ts_tablet_manager.cc:1888] T 1d98617b2344473d83eb448adee33dcd P 1ade0354c44c4c37ae480c7a94b11e1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:54.832644 4199 log.cc:1196] T 1d98617b2344473d83eb448adee33dcd P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal/wals/1d98617b2344473d83eb448adee33dcd
I20240502 21:26:54.833016 4199 ts_tablet_manager.cc:1909] T 1d98617b2344473d83eb448adee33dcd P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting consensus metadata
I20240502 21:26:54.833819 31381 catalog_manager.cc:4830] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): tablet 1d98617b2344473d83eb448adee33dcd (table rebalancer_dynamic_table_new_name [id=ed617dcec8e44a93a1d030202f77c0b6]) successfully deleted
I20240502 21:26:54.837996 4198 ts_tablet_manager.cc:1888] T 1d98617b2344473d83eb448adee33dcd P 9dec645b473f40578e726f3e50e219bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:54.838135 4198 log.cc:1196] T 1d98617b2344473d83eb448adee33dcd P 9dec645b473f40578e726f3e50e219bf: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal/wals/1d98617b2344473d83eb448adee33dcd
I20240502 21:26:54.838559 4198 ts_tablet_manager.cc:1909] T 1d98617b2344473d83eb448adee33dcd P 9dec645b473f40578e726f3e50e219bf: Deleting consensus metadata
I20240502 21:26:54.840080 31379 catalog_manager.cc:4830] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): tablet 1d98617b2344473d83eb448adee33dcd (table rebalancer_dynamic_table_new_name [id=ed617dcec8e44a93a1d030202f77c0b6]) successfully deleted
I20240502 21:26:54.843575 1833 consensus_queue.cc:237] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } }
I20240502 21:26:54.845852 2097 raft_consensus.cc:1261] T 577e7a586ad9459f8e77ce4b8f563c09 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:54.845852 2225 raft_consensus.cc:1261] T 577e7a586ad9459f8e77ce4b8f563c09 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:26:54.845983 1770 consensus_peers.cc:485] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db -> Peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): Couldn't send request to peer 05f4fac41c4f4959b9b316cb112d0ada. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 577e7a586ad9459f8e77ce4b8f563c09. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:54.846549 4118 consensus_queue.cc:1035] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:54.846871 3529 consensus_queue.cc:1035] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:54.850122 3669 raft_consensus.cc:2932] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } } }
I20240502 21:26:54.850594 2225 raft_consensus.cc:2932] T 577e7a586ad9459f8e77ce4b8f563c09 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } } }
I20240502 21:26:54.851336 2097 raft_consensus.cc:2932] T 577e7a586ad9459f8e77ce4b8f563c09 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } } }
I20240502 21:26:54.855245 31393 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_extra_table_24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:54.857081 31387 catalog_manager.cc:5484] T 577e7a586ad9459f8e77ce4b8f563c09 P d1406e92aec84ac585304e57bd37a40d reported cstate change: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } } }
I20240502 21:26:54.876186 31516 tablet_service.cc:1467] Processing CreateTablet for tablet a699f448e116428f85cfee6ccbc98612 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=da186563aba24f6d860086d2a27eeff3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:54.876641 31516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a699f448e116428f85cfee6ccbc98612. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:54.878115 2076 tablet_service.cc:1467] Processing CreateTablet for tablet 4ed9c8356c4c4a5e94b755d58b3eb7a0 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=da186563aba24f6d860086d2a27eeff3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:54.878564 2076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ed9c8356c4c4a5e94b755d58b3eb7a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:54.880728 2210 tablet_service.cc:1467] Processing CreateTablet for tablet a699f448e116428f85cfee6ccbc98612 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=da186563aba24f6d860086d2a27eeff3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:54.880995 2210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a699f448e116428f85cfee6ccbc98612. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:54.883973 31775 tablet_service.cc:1467] Processing CreateTablet for tablet a699f448e116428f85cfee6ccbc98612 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=da186563aba24f6d860086d2a27eeff3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:54.884367 31775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a699f448e116428f85cfee6ccbc98612. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:54.888685 3765 raft_consensus.cc:1050] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d: attempting to promote NON_VOTER 08d7a0680d244b55b93d17f9ab57b84c to VOTER
I20240502 21:26:54.889611 2208 tablet_service.cc:1467] Processing CreateTablet for tablet 4ed9c8356c4c4a5e94b755d58b3eb7a0 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=da186563aba24f6d860086d2a27eeff3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:54.889986 2208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ed9c8356c4c4a5e94b755d58b3eb7a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:54.889786 3765 consensus_queue.cc:237] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } }
I20240502 21:26:54.891557 1813 tablet_service.cc:1467] Processing CreateTablet for tablet b8db5de7df864ab983504e96fb738106 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=da186563aba24f6d860086d2a27eeff3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:54.892052 1813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8db5de7df864ab983504e96fb738106. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:54.893106 1946 tablet_service.cc:1467] Processing CreateTablet for tablet 4ed9c8356c4c4a5e94b755d58b3eb7a0 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=da186563aba24f6d860086d2a27eeff3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:54.893500 1946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ed9c8356c4c4a5e94b755d58b3eb7a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:54.898162 1945 tablet_service.cc:1467] Processing CreateTablet for tablet b8db5de7df864ab983504e96fb738106 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=da186563aba24f6d860086d2a27eeff3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:54.898628 1945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8db5de7df864ab983504e96fb738106. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:54.908306 31774 tablet_service.cc:1467] Processing CreateTablet for tablet b8db5de7df864ab983504e96fb738106 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=da186563aba24f6d860086d2a27eeff3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:54.908722 31774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8db5de7df864ab983504e96fb738106. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:54.917484 31387 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:26:54.935734 2077 tablet_service.cc:1467] Processing CreateTablet for tablet 43c02342d1ce4b53870e0495116ef173 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=03638b9bda5347dcbf84b97a9589a9c6]), partition=
I20240502 21:26:54.935933 1945 tablet_service.cc:1467] Processing CreateTablet for tablet 43c02342d1ce4b53870e0495116ef173 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=03638b9bda5347dcbf84b97a9589a9c6]), partition=
I20240502 21:26:54.936064 2077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43c02342d1ce4b53870e0495116ef173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:54.936323 1945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43c02342d1ce4b53870e0495116ef173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:54.941275 2210 tablet_service.cc:1467] Processing CreateTablet for tablet 43c02342d1ce4b53870e0495116ef173 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=03638b9bda5347dcbf84b97a9589a9c6]), partition=
I20240502 21:26:54.941618 2210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43c02342d1ce4b53870e0495116ef173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:54.946141 4211 tablet_bootstrap.cc:492] T a699f448e116428f85cfee6ccbc98612 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:54.949290 4211 tablet_bootstrap.cc:654] T a699f448e116428f85cfee6ccbc98612 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:54.950930 4214 tablet_bootstrap.cc:492] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:54.953133 4214 tablet_bootstrap.cc:654] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:54.955363 4209 tablet_bootstrap.cc:492] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:54.956949 4212 tablet_bootstrap.cc:492] T a699f448e116428f85cfee6ccbc98612 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:54.957846 4216 ts_tablet_manager.cc:902] T 0b8d119ee8474a40ac08df61df5b8423 P 9dec645b473f40578e726f3e50e219bf: Initiating tablet copy from peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:54.959231 4212 tablet_bootstrap.cc:654] T a699f448e116428f85cfee6ccbc98612 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:54.954499 4213 tablet_bootstrap.cc:492] T b8db5de7df864ab983504e96fb738106 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:54.962926 4213 tablet_bootstrap.cc:654] T b8db5de7df864ab983504e96fb738106 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:54.966230 31796 raft_consensus.cc:1261] T f5e82c0ab5c441efbcfbcd383b8db82f P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEARNER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:54.966751 2891 consensus_queue.cc:1035] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:54.969400 4209 tablet_bootstrap.cc:654] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:54.975507 4211 tablet_bootstrap.cc:492] T a699f448e116428f85cfee6ccbc98612 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:54.975703 4211 ts_tablet_manager.cc:1367] T a699f448e116428f85cfee6ccbc98612 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20240502 21:26:54.976990 4211 raft_consensus.cc:348] T a699f448e116428f85cfee6ccbc98612 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:54.977303 4211 raft_consensus.cc:374] T a699f448e116428f85cfee6ccbc98612 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:54.977440 4211 raft_consensus.cc:729] T a699f448e116428f85cfee6ccbc98612 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:54.977699 4211 consensus_queue.cc:260] T a699f448e116428f85cfee6ccbc98612 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:54.981443 4214 tablet_bootstrap.cc:492] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:54.981669 4214 ts_tablet_manager.cc:1367] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20240502 21:26:54.982452 4214 raft_consensus.cc:348] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:54.982726 4214 raft_consensus.cc:374] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:54.982947 4214 raft_consensus.cc:729] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:54.983247 4214 consensus_queue.cc:260] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:54.985953 31923 consensus_queue.cc:237] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } }
I20240502 21:26:54.998001 1833 raft_consensus.cc:1261] T f5e82c0ab5c441efbcfbcd383b8db82f P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:54.998696 2891 consensus_queue.cc:1035] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:55.001529 4210 tablet_bootstrap.cc:492] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:55.003273 3951 raft_consensus.cc:482] T a699f448e116428f85cfee6ccbc98612 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:55.003422 3951 raft_consensus.cc:504] T a699f448e116428f85cfee6ccbc98612 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:55.003690 4210 tablet_bootstrap.cc:654] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:55.004107 3951 leader_election.cc:288] T a699f448e116428f85cfee6ccbc98612 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:55.004870 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a699f448e116428f85cfee6ccbc98612" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:55.005419 4213 tablet_bootstrap.cc:492] T b8db5de7df864ab983504e96fb738106 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:55.005647 4213 ts_tablet_manager.cc:1367] T b8db5de7df864ab983504e96fb738106 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.051s user 0.000s sys 0.004s
W20240502 21:26:55.006232 31733 leader_election.cc:341] T a699f448e116428f85cfee6ccbc98612 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:55.006582 4213 raft_consensus.cc:348] T b8db5de7df864ab983504e96fb738106 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:55.006836 4213 raft_consensus.cc:374] T b8db5de7df864ab983504e96fb738106 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:55.006798 2225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a699f448e116428f85cfee6ccbc98612" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:26:55.007004 4213 raft_consensus.cc:729] T b8db5de7df864ab983504e96fb738106 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
W20240502 21:26:55.007323 31732 leader_election.cc:341] T a699f448e116428f85cfee6ccbc98612 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:55.007443 31732 leader_election.cc:302] T a699f448e116428f85cfee6ccbc98612 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c; no voters: a44f41b0b6644fb6aa29476aed8c964f, d1406e92aec84ac585304e57bd37a40d
I20240502 21:26:55.007277 4213 consensus_queue.cc:260] T b8db5de7df864ab983504e96fb738106 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:55.007787 3951 raft_consensus.cc:2726] T a699f448e116428f85cfee6ccbc98612 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:26:55.013540 4209 tablet_bootstrap.cc:492] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:55.013772 4209 ts_tablet_manager.cc:1367] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.059s user 0.004s sys 0.000s
I20240502 21:26:55.014844 4209 raft_consensus.cc:348] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:55.015228 4209 raft_consensus.cc:374] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:55.015383 4209 raft_consensus.cc:729] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:55.015687 4209 consensus_queue.cc:260] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:55.016495 4209 ts_tablet_manager.cc:1398] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20240502 21:26:55.018539 31862 consensus_peers.cc:485] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 -> Peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): Couldn't send request to peer 1e04157b98dd4fad98858c290b73c3db. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 83904952a5a541ecac508449e23e70dd. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:55.018580 31797 raft_consensus.cc:1261] T 83904952a5a541ecac508449e23e70dd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:26:55.019791 3824 consensus_queue.cc:1035] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:26:55.024531 4078 raft_consensus.cc:2932] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } } }
I20240502 21:26:55.027259 4211 ts_tablet_manager.cc:1398] T a699f448e116428f85cfee6ccbc98612 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.051s user 0.002s sys 0.000s
I20240502 21:26:55.028867 4223 ts_tablet_manager.cc:902] T 1022c4c6ee8642ca95250933a940fe7a P 05f4fac41c4f4959b9b316cb112d0ada: Initiating tablet copy from peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:55.032893 31395 catalog_manager.cc:5484] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: config changed from index 6 to 7, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:55.033082 31797 raft_consensus.cc:2932] T 83904952a5a541ecac508449e23e70dd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } } }
I20240502 21:26:55.039199 4213 ts_tablet_manager.cc:1398] T b8db5de7df864ab983504e96fb738106 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.033s user 0.000s sys 0.002s
I20240502 21:26:55.044271 4211 tablet_bootstrap.cc:492] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:55.048686 4214 ts_tablet_manager.cc:1398] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:55.049104 4214 tablet_bootstrap.cc:492] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:55.049259 1833 raft_consensus.cc:2932] T f5e82c0ab5c441efbcfbcd383b8db82f P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:26:55.051482 4211 tablet_bootstrap.cc:654] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:55.053247 4212 tablet_bootstrap.cc:492] T a699f448e116428f85cfee6ccbc98612 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:55.053444 4212 ts_tablet_manager.cc:1367] T a699f448e116428f85cfee6ccbc98612 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.097s user 0.004s sys 0.000s
I20240502 21:26:55.054401 4212 raft_consensus.cc:348] T a699f448e116428f85cfee6ccbc98612 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:55.054630 4235 ts_tablet_manager.cc:902] T 46279270b2334633906af57c4b43e969 P 05f4fac41c4f4959b9b316cb112d0ada: Initiating tablet copy from peer 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:55.054903 4212 raft_consensus.cc:374] T a699f448e116428f85cfee6ccbc98612 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:55.056104 4212 raft_consensus.cc:729] T a699f448e116428f85cfee6ccbc98612 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:55.056130 4216 tablet_copy_client.cc:308] T 0b8d119ee8474a40ac08df61df5b8423 P 9dec645b473f40578e726f3e50e219bf: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.130:37645
I20240502 21:26:55.056872 4212 consensus_queue.cc:260] T a699f448e116428f85cfee6ccbc98612 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:55.060575 31686 tablet_copy_service.cc:140] P 05f4fac41c4f4959b9b316cb112d0ada: Received BeginTabletCopySession request for tablet 0b8d119ee8474a40ac08df61df5b8423 from peer 9dec645b473f40578e726f3e50e219bf ({username='slave'} at 127.24.226.134:40513)
I20240502 21:26:55.060745 31686 tablet_copy_service.cc:161] P 05f4fac41c4f4959b9b316cb112d0ada: Beginning new tablet copy session on tablet 0b8d119ee8474a40ac08df61df5b8423 from peer 9dec645b473f40578e726f3e50e219bf at {username='slave'} at 127.24.226.134:40513: session id = 9dec645b473f40578e726f3e50e219bf-0b8d119ee8474a40ac08df61df5b8423
I20240502 21:26:55.064684 31686 tablet_copy_source_session.cc:205] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:55.066607 4216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b8d119ee8474a40ac08df61df5b8423. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:55.068944 3529 raft_consensus.cc:482] T b8db5de7df864ab983504e96fb738106 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:55.068998 4212 ts_tablet_manager.cc:1398] T a699f448e116428f85cfee6ccbc98612 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240502 21:26:55.069115 3529 raft_consensus.cc:504] T b8db5de7df864ab983504e96fb738106 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:55.069912 3529 leader_election.cc:288] T b8db5de7df864ab983504e96fb738106 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:55.044880 3765 raft_consensus.cc:2932] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:26:55.070376 4209 tablet_bootstrap.cc:492] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:55.072012 4209 tablet_bootstrap.cc:654] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:55.049669 2097 raft_consensus.cc:1261] T 83904952a5a541ecac508449e23e70dd P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:26:55.050720 2094 raft_consensus.cc:1261] T f5e82c0ab5c441efbcfbcd383b8db82f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:55.072955 4216 tablet_copy_client.cc:783] T 0b8d119ee8474a40ac08df61df5b8423 P 9dec645b473f40578e726f3e50e219bf: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:55.073156 4216 tablet_copy_client.cc:647] T 0b8d119ee8474a40ac08df61df5b8423 P 9dec645b473f40578e726f3e50e219bf: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:55.073334 4078 consensus_queue.cc:1035] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:26:55.073755 3769 consensus_queue.cc:1035] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:55.074337 4214 tablet_bootstrap.cc:654] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:55.075881 1964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8db5de7df864ab983504e96fb738106" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:26:55.077129 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8db5de7df864ab983504e96fb738106" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
W20240502 21:26:55.077819 1769 leader_election.cc:341] T b8db5de7df864ab983504e96fb738106 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:26:55.078393 1767 leader_election.cc:341] T b8db5de7df864ab983504e96fb738106 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:55.078521 1767 leader_election.cc:302] T b8db5de7df864ab983504e96fb738106 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db; no voters: 08d7a0680d244b55b93d17f9ab57b84c, 9dec645b473f40578e726f3e50e219bf
I20240502 21:26:55.079134 4118 raft_consensus.cc:2726] T b8db5de7df864ab983504e96fb738106 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:26:55.082728 31796 raft_consensus.cc:2932] T f5e82c0ab5c441efbcfbcd383b8db82f P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:26:55.086028 4211 tablet_bootstrap.cc:492] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:55.086298 4211 ts_tablet_manager.cc:1367] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.000s
I20240502 21:26:55.086608 4216 tablet_copy_client.cc:523] T 0b8d119ee8474a40ac08df61df5b8423 P 9dec645b473f40578e726f3e50e219bf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:55.087221 4211 raft_consensus.cc:348] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:55.087520 4211 raft_consensus.cc:374] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:55.087630 4211 raft_consensus.cc:729] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:55.087921 4211 consensus_queue.cc:260] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:55.088738 4211 ts_tablet_manager.cc:1398] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:55.090304 4214 tablet_bootstrap.cc:492] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:55.091287 4216 tablet_bootstrap.cc:492] T 0b8d119ee8474a40ac08df61df5b8423 P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:55.092116 4210 tablet_bootstrap.cc:492] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:26:55.092268 4210 ts_tablet_manager.cc:1367] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.091s user 0.002s sys 0.002s
I20240502 21:26:55.093202 4210 raft_consensus.cc:348] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:55.093401 4210 raft_consensus.cc:374] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:55.093456 4214 ts_tablet_manager.cc:1367] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.000s
I20240502 21:26:55.093542 4210 raft_consensus.cc:729] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:26:55.093814 4210 consensus_queue.cc:260] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:55.094620 4214 raft_consensus.cc:348] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:55.095804 4210 ts_tablet_manager.cc:1398] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:26:55.096930 4210 tablet_bootstrap.cc:492] T b8db5de7df864ab983504e96fb738106 P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:55.107070 4214 raft_consensus.cc:374] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:55.107422 4214 raft_consensus.cc:729] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:55.109310 4210 tablet_bootstrap.cc:654] T b8db5de7df864ab983504e96fb738106 P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:55.111711 2097 raft_consensus.cc:2932] T f5e82c0ab5c441efbcfbcd383b8db82f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:26:55.113312 4249 ts_tablet_manager.cc:902] T b20df8896a2b423cb53c2c1a2a1b98e2 P d1406e92aec84ac585304e57bd37a40d: Initiating tablet copy from peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:55.113847 4214 consensus_queue.cc:260] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:55.116046 4214 ts_tablet_manager.cc:1398] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240502 21:26:55.116573 4248 ts_tablet_manager.cc:902] T de2216a74f8344779a881651b66a85e5 P 08d7a0680d244b55b93d17f9ab57b84c: Initiating tablet copy from peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:55.117888 31394 catalog_manager.cc:5484] T f5e82c0ab5c441efbcfbcd383b8db82f P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index 2 to 3, 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:26:55.119081 4209 tablet_bootstrap.cc:492] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:55.119122 2094 raft_consensus.cc:2932] T 83904952a5a541ecac508449e23e70dd P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } } }
I20240502 21:26:55.119578 4209 ts_tablet_manager.cc:1367] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.049s user 0.000s sys 0.006s
I20240502 21:26:55.122305 4210 tablet_bootstrap.cc:492] T b8db5de7df864ab983504e96fb738106 P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:26:55.122534 4210 ts_tablet_manager.cc:1367] T b8db5de7df864ab983504e96fb738106 P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.002s
I20240502 21:26:55.123301 4210 raft_consensus.cc:348] T b8db5de7df864ab983504e96fb738106 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:55.123572 4210 raft_consensus.cc:374] T b8db5de7df864ab983504e96fb738106 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:55.123667 4210 raft_consensus.cc:729] T b8db5de7df864ab983504e96fb738106 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:26:55.123932 4210 consensus_queue.cc:260] T b8db5de7df864ab983504e96fb738106 P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:55.124712 4210 ts_tablet_manager.cc:1398] T b8db5de7df864ab983504e96fb738106 P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:26:55.125232 4210 tablet_bootstrap.cc:492] T 43c02342d1ce4b53870e0495116ef173 P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:55.120882 4209 raft_consensus.cc:348] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:55.126503 4209 raft_consensus.cc:374] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:55.126616 4209 raft_consensus.cc:729] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:55.126837 4210 tablet_bootstrap.cc:654] T 43c02342d1ce4b53870e0495116ef173 P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:55.126912 4209 consensus_queue.cc:260] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:55.133471 4209 ts_tablet_manager.cc:1398] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240502 21:26:55.134006 4209 tablet_bootstrap.cc:492] T 43c02342d1ce4b53870e0495116ef173 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:55.135794 4209 tablet_bootstrap.cc:654] T 43c02342d1ce4b53870e0495116ef173 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:55.139022 4210 tablet_bootstrap.cc:492] T 43c02342d1ce4b53870e0495116ef173 P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:26:55.139194 4210 ts_tablet_manager.cc:1367] T 43c02342d1ce4b53870e0495116ef173 P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240502 21:26:55.139912 4210 raft_consensus.cc:348] T 43c02342d1ce4b53870e0495116ef173 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:55.140168 4210 raft_consensus.cc:374] T 43c02342d1ce4b53870e0495116ef173 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:55.141481 4210 raft_consensus.cc:729] T 43c02342d1ce4b53870e0495116ef173 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:26:55.141724 4210 consensus_queue.cc:260] T 43c02342d1ce4b53870e0495116ef173 P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:55.150873 2230 consensus_queue.cc:237] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } }
I20240502 21:26:55.148605 3818 raft_consensus.cc:482] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:55.152561 3818 raft_consensus.cc:504] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:55.153136 3818 leader_election.cc:288] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:55.153810 4210 ts_tablet_manager.cc:1398] T 43c02342d1ce4b53870e0495116ef173 P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20240502 21:26:55.153852 2094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed9c8356c4c4a5e94b755d58b3eb7a0" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:55.154187 2094 raft_consensus.cc:2454] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 0.
I20240502 21:26:55.154657 1901 leader_election.cc:302] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:26:55.154983 3818 raft_consensus.cc:2781] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:55.155136 3818 raft_consensus.cc:482] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:55.155252 3818 raft_consensus.cc:3037] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:55.158061 31797 raft_consensus.cc:1261] T f5e82c0ab5c441efbcfbcd383b8db82f P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:55.159998 3818 raft_consensus.cc:504] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:55.161166 2094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed9c8356c4c4a5e94b755d58b3eb7a0" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:26:55.161371 3818 leader_election.cc:288] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 election: Requested vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:55.161438 2094 raft_consensus.cc:3037] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:55.162272 2891 consensus_queue.cc:1035] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:55.164462 2094 raft_consensus.cc:2454] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 1.
I20240502 21:26:55.164952 1901 leader_election.cc:302] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:26:55.165483 3818 raft_consensus.cc:2781] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:55.171869 4209 tablet_bootstrap.cc:492] T 43c02342d1ce4b53870e0495116ef173 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:55.172058 4209 ts_tablet_manager.cc:1367] T 43c02342d1ce4b53870e0495116ef173 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.038s user 0.001s sys 0.004s
I20240502 21:26:55.172585 2230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed9c8356c4c4a5e94b755d58b3eb7a0" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:26:55.172780 2230 raft_consensus.cc:2454] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 0.
I20240502 21:26:55.173384 4248 tablet_copy_client.cc:308] T de2216a74f8344779a881651b66a85e5 P 08d7a0680d244b55b93d17f9ab57b84c: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.130:37645
I20240502 21:26:55.174024 2230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed9c8356c4c4a5e94b755d58b3eb7a0" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:26:55.174127 31686 tablet_copy_service.cc:140] P 05f4fac41c4f4959b9b316cb112d0ada: Received BeginTabletCopySession request for tablet de2216a74f8344779a881651b66a85e5 from peer 08d7a0680d244b55b93d17f9ab57b84c ({username='slave'} at 127.24.226.131:52471)
I20240502 21:26:55.174202 2230 raft_consensus.cc:3037] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:55.174297 31686 tablet_copy_service.cc:161] P 05f4fac41c4f4959b9b316cb112d0ada: Beginning new tablet copy session on tablet de2216a74f8344779a881651b66a85e5 from peer 08d7a0680d244b55b93d17f9ab57b84c at {username='slave'} at 127.24.226.131:52471: session id = 08d7a0680d244b55b93d17f9ab57b84c-de2216a74f8344779a881651b66a85e5
I20240502 21:26:55.176879 2230 raft_consensus.cc:2454] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 1.
I20240502 21:26:55.178190 2891 raft_consensus.cc:2932] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:26:55.180410 3769 raft_consensus.cc:482] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:55.180584 3769 raft_consensus.cc:504] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:55.181174 3769 leader_election.cc:288] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:55.181588 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a699f448e116428f85cfee6ccbc98612" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:55.181701 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a699f448e116428f85cfee6ccbc98612" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:55.181820 31536 raft_consensus.cc:2454] T a699f448e116428f85cfee6ccbc98612 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:26:55.181890 31797 raft_consensus.cc:2454] T a699f448e116428f85cfee6ccbc98612 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:26:55.182479 2164 leader_election.cc:302] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a44f41b0b6644fb6aa29476aed8c964f, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:55.182881 31686 tablet_copy_source_session.cc:205] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:55.182955 3769 raft_consensus.cc:2781] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:55.183063 3769 raft_consensus.cc:482] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:55.183128 3769 raft_consensus.cc:3037] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:55.186518 4248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de2216a74f8344779a881651b66a85e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:55.186789 4216 tablet_bootstrap.cc:492] T 0b8d119ee8474a40ac08df61df5b8423 P 9dec645b473f40578e726f3e50e219bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:55.187574 3818 raft_consensus.cc:686] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 9dec645b473f40578e726f3e50e219bf [term 1 LEADER]: Becoming Leader. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Running, Role: LEADER
I20240502 21:26:55.187862 3818 consensus_queue.cc:237] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:55.188333 4216 tablet_bootstrap.cc:492] T 0b8d119ee8474a40ac08df61df5b8423 P 9dec645b473f40578e726f3e50e219bf: Bootstrap complete.
I20240502 21:26:55.188623 4216 ts_tablet_manager.cc:1367] T 0b8d119ee8474a40ac08df61df5b8423 P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.097s user 0.043s sys 0.008s
I20240502 21:26:55.189474 4216 raft_consensus.cc:348] T 0b8d119ee8474a40ac08df61df5b8423 P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:26:55.189718 4216 raft_consensus.cc:729] T 0b8d119ee8474a40ac08df61df5b8423 P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: LEARNER
I20240502 21:26:55.189899 4216 consensus_queue.cc:260] T 0b8d119ee8474a40ac08df61df5b8423 P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:26:55.190688 4216 ts_tablet_manager.cc:1398] T 0b8d119ee8474a40ac08df61df5b8423 P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:55.192684 4123 raft_consensus.cc:482] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:55.192837 4123 raft_consensus.cc:504] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:55.193356 31686 tablet_copy_service.cc:342] P 05f4fac41c4f4959b9b316cb112d0ada: Request end of tablet copy session 9dec645b473f40578e726f3e50e219bf-0b8d119ee8474a40ac08df61df5b8423 received from {username='slave'} at 127.24.226.134:40513
I20240502 21:26:55.193488 31686 tablet_copy_service.cc:433] P 05f4fac41c4f4959b9b316cb112d0ada: ending tablet copy session 9dec645b473f40578e726f3e50e219bf-0b8d119ee8474a40ac08df61df5b8423 on tablet 0b8d119ee8474a40ac08df61df5b8423 with peer 9dec645b473f40578e726f3e50e219bf
I20240502 21:26:55.193572 4123 leader_election.cc:288] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:55.194228 1966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43c02342d1ce4b53870e0495116ef173" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:26:55.194449 1966 raft_consensus.cc:2454] T 43c02342d1ce4b53870e0495116ef173 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 0.
I20240502 21:26:55.195168 2030 leader_election.cc:302] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:26:55.195561 4123 raft_consensus.cc:2781] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:55.195660 4123 raft_consensus.cc:482] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:55.195755 4123 raft_consensus.cc:3037] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:55.197255 3951 raft_consensus.cc:482] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:55.197470 3951 raft_consensus.cc:504] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:55.198359 3951 leader_election.cc:288] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:26:55.199002 4123 raft_consensus.cc:504] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:55.199061 1832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8db5de7df864ab983504e96fb738106" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:26:55.199316 1832 raft_consensus.cc:2454] T b8db5de7df864ab983504e96fb738106 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:55.199725 1966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8db5de7df864ab983504e96fb738106" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:26:55.199914 1966 raft_consensus.cc:2454] T b8db5de7df864ab983504e96fb738106 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:26:55.200116 1965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43c02342d1ce4b53870e0495116ef173" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf"
I20240502 21:26:55.200381 1965 raft_consensus.cc:3037] T 43c02342d1ce4b53870e0495116ef173 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:55.200780 31732 leader_election.cc:302] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 1e04157b98dd4fad98858c290b73c3db; no voters:
I20240502 21:26:55.201251 4167 raft_consensus.cc:2781] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:55.201370 4167 raft_consensus.cc:482] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:55.201478 4167 raft_consensus.cc:3037] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:55.203251 1965 raft_consensus.cc:2454] T 43c02342d1ce4b53870e0495116ef173 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 1.
I20240502 21:26:55.204716 2030 leader_election.cc:302] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:26:55.204991 4167 raft_consensus.cc:504] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:55.207302 4123 leader_election.cc:288] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 election: Requested vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:55.207520 4123 raft_consensus.cc:2781] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:55.207649 4123 raft_consensus.cc:686] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEADER]: Becoming Leader. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Running, Role: LEADER
I20240502 21:26:55.207871 1832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8db5de7df864ab983504e96fb738106" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:26:55.208076 1832 raft_consensus.cc:3037] T b8db5de7df864ab983504e96fb738106 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:55.208295 4167 leader_election.cc:288] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Requested vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:26:55.209120 4216 ts_tablet_manager.cc:902] T 182b3f15c06c4c07ade0a8854d3ecafd P 9dec645b473f40578e726f3e50e219bf: Initiating tablet copy from peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:55.209419 1965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8db5de7df864ab983504e96fb738106" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf"
I20240502 21:26:55.207962 4123 consensus_queue.cc:237] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:55.209932 1965 raft_consensus.cc:3037] T b8db5de7df864ab983504e96fb738106 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:55.212843 1965 raft_consensus.cc:2454] T b8db5de7df864ab983504e96fb738106 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:55.213204 31730 leader_election.cc:302] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:26:55.213465 3951 raft_consensus.cc:2781] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:55.213608 3951 raft_consensus.cc:686] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Becoming Leader. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Running, Role: LEADER
I20240502 21:26:55.213835 3951 consensus_queue.cc:237] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:55.215049 1832 raft_consensus.cc:2454] T b8db5de7df864ab983504e96fb738106 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:26:55.215617 1834 raft_consensus.cc:1261] T f5e82c0ab5c441efbcfbcd383b8db82f P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:55.219393 31394 catalog_manager.cc:5484] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: term changed from 0 to 1, leader changed from <none> to 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131). New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:55.225100 2230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43c02342d1ce4b53870e0495116ef173" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:26:55.225750 3765 consensus_queue.cc:1035] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:55.225909 2225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43c02342d1ce4b53870e0495116ef173" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:26:55.226629 3769 raft_consensus.cc:504] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:55.227986 1834 raft_consensus.cc:2932] T f5e82c0ab5c441efbcfbcd383b8db82f P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:26:55.228379 3769 leader_election.cc:288] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Requested vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:55.229688 31381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f5e82c0ab5c441efbcfbcd383b8db82f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:55.230115 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a699f448e116428f85cfee6ccbc98612" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:26:55.230341 31536 raft_consensus.cc:3037] T a699f448e116428f85cfee6ccbc98612 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:55.232668 4209 raft_consensus.cc:348] T 43c02342d1ce4b53870e0495116ef173 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:55.232856 4209 raft_consensus.cc:374] T 43c02342d1ce4b53870e0495116ef173 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:55.233000 4209 raft_consensus.cc:729] T 43c02342d1ce4b53870e0495116ef173 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:55.233345 4209 consensus_queue.cc:260] T 43c02342d1ce4b53870e0495116ef173 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:55.235180 31394 catalog_manager.cc:5484] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d reported cstate change: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:55.235915 31536 raft_consensus.cc:2454] T a699f448e116428f85cfee6ccbc98612 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 1.
I20240502 21:26:55.236446 2164 leader_election.cc:302] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a44f41b0b6644fb6aa29476aed8c964f, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:55.236572 4249 tablet_copy_client.cc:308] T b20df8896a2b423cb53c2c1a2a1b98e2 P d1406e92aec84ac585304e57bd37a40d: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.129:41039
I20240502 21:26:55.236828 3765 raft_consensus.cc:2781] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:55.236951 3765 raft_consensus.cc:686] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Becoming Leader. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Running, Role: LEADER
I20240502 21:26:55.237229 3765 consensus_queue.cc:237] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:55.237751 31556 tablet_copy_service.cc:140] P a44f41b0b6644fb6aa29476aed8c964f: Received BeginTabletCopySession request for tablet b20df8896a2b423cb53c2c1a2a1b98e2 from peer d1406e92aec84ac585304e57bd37a40d ({username='slave'} at 127.24.226.136:52831)
I20240502 21:26:55.237985 31556 tablet_copy_service.cc:161] P a44f41b0b6644fb6aa29476aed8c964f: Beginning new tablet copy session on tablet b20df8896a2b423cb53c2c1a2a1b98e2 from peer d1406e92aec84ac585304e57bd37a40d at {username='slave'} at 127.24.226.136:52831: session id = d1406e92aec84ac585304e57bd37a40d-b20df8896a2b423cb53c2c1a2a1b98e2
I20240502 21:26:55.238337 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a699f448e116428f85cfee6ccbc98612" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:55.238564 31797 raft_consensus.cc:3037] T a699f448e116428f85cfee6ccbc98612 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:55.240136 4209 ts_tablet_manager.cc:1398] T 43c02342d1ce4b53870e0495116ef173 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.062s user 0.002s sys 0.000s
I20240502 21:26:55.241708 31797 raft_consensus.cc:2454] T a699f448e116428f85cfee6ccbc98612 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 1.
I20240502 21:26:55.246704 4275 ts_tablet_manager.cc:902] T 577e7a586ad9459f8e77ce4b8f563c09 P 05f4fac41c4f4959b9b316cb112d0ada: Initiating tablet copy from peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:26:55.250667 4248 tablet_copy_client.cc:783] T de2216a74f8344779a881651b66a85e5 P 08d7a0680d244b55b93d17f9ab57b84c: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:55.250849 31556 tablet_copy_source_session.cc:205] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:55.251031 4248 tablet_copy_client.cc:647] T de2216a74f8344779a881651b66a85e5 P 08d7a0680d244b55b93d17f9ab57b84c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:55.252856 4248 tablet_copy_client.cc:523] T de2216a74f8344779a881651b66a85e5 P 08d7a0680d244b55b93d17f9ab57b84c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:55.255476 2225 consensus_queue.cc:237] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } }
I20240502 21:26:55.259622 2076 tablet_service.cc:1514] Processing DeleteTablet for tablet f5e82c0ab5c441efbcfbcd383b8db82f with delete_type TABLET_DATA_TOMBSTONED (TS 1ade0354c44c4c37ae480c7a94b11e1b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36740
I20240502 21:26:55.260000 4199 tablet_replica.cc:330] T f5e82c0ab5c441efbcfbcd383b8db82f P 1ade0354c44c4c37ae480c7a94b11e1b: stopping tablet replica
I20240502 21:26:55.260246 4199 raft_consensus.cc:2229] T f5e82c0ab5c441efbcfbcd383b8db82f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:55.260465 4199 raft_consensus.cc:2258] T f5e82c0ab5c441efbcfbcd383b8db82f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:55.257879 31796 raft_consensus.cc:2932] T f5e82c0ab5c441efbcfbcd383b8db82f P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:26:55.261317 4199 ts_tablet_manager.cc:1875] T f5e82c0ab5c441efbcfbcd383b8db82f P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:55.262302 4248 tablet_bootstrap.cc:492] T de2216a74f8344779a881651b66a85e5 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:55.267474 4199 ts_tablet_manager.cc:1888] T f5e82c0ab5c441efbcfbcd383b8db82f P 1ade0354c44c4c37ae480c7a94b11e1b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:26:55.267699 4199 log.cc:1196] T f5e82c0ab5c441efbcfbcd383b8db82f P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal/wals/f5e82c0ab5c441efbcfbcd383b8db82f
I20240502 21:26:55.268617 31381 catalog_manager.cc:4830] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): tablet f5e82c0ab5c441efbcfbcd383b8db82f (table rebalancer_extra_table_8 [id=4cfb4e8589a24f279c0e7f552012212c]) successfully deleted
W20240502 21:26:55.305876 2032 leader_election.cc:341] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:26:55.306361 2032 leader_election.cc:341] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:55.306959 4223 tablet_copy_client.cc:308] T 1022c4c6ee8642ca95250933a940fe7a P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.136:34907
I20240502 21:26:55.307740 2251 tablet_copy_service.cc:140] P d1406e92aec84ac585304e57bd37a40d: Received BeginTabletCopySession request for tablet 1022c4c6ee8642ca95250933a940fe7a from peer 05f4fac41c4f4959b9b316cb112d0ada ({username='slave'} at 127.24.226.130:56069)
I20240502 21:26:55.307915 2251 tablet_copy_service.cc:161] P d1406e92aec84ac585304e57bd37a40d: Beginning new tablet copy session on tablet 1022c4c6ee8642ca95250933a940fe7a from peer 05f4fac41c4f4959b9b316cb112d0ada at {username='slave'} at 127.24.226.130:56069: session id = 05f4fac41c4f4959b9b316cb112d0ada-1022c4c6ee8642ca95250933a940fe7a
I20240502 21:26:55.308267 4286 ts_tablet_manager.cc:902] T 1ed225bdefc64ddd8eef2109bfd969df P d1406e92aec84ac585304e57bd37a40d: Initiating tablet copy from peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:55.310027 1833 raft_consensus.cc:1261] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:55.310587 2891 consensus_queue.cc:1035] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:55.311975 1964 raft_consensus.cc:1261] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:55.312551 4216 tablet_copy_client.cc:308] T 182b3f15c06c4c07ade0a8854d3ecafd P 9dec645b473f40578e726f3e50e219bf: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.130:37645
I20240502 21:26:55.312674 2891 consensus_queue.cc:1035] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:55.313225 2251 tablet_copy_source_session.cc:205] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:55.313328 31686 tablet_copy_service.cc:140] P 05f4fac41c4f4959b9b316cb112d0ada: Received BeginTabletCopySession request for tablet 182b3f15c06c4c07ade0a8854d3ecafd from peer 9dec645b473f40578e726f3e50e219bf ({username='slave'} at 127.24.226.134:40513)
I20240502 21:26:55.313519 31686 tablet_copy_service.cc:161] P 05f4fac41c4f4959b9b316cb112d0ada: Beginning new tablet copy session on tablet 182b3f15c06c4c07ade0a8854d3ecafd from peer 9dec645b473f40578e726f3e50e219bf at {username='slave'} at 127.24.226.134:40513: session id = 9dec645b473f40578e726f3e50e219bf-182b3f15c06c4c07ade0a8854d3ecafd
I20240502 21:26:55.315325 31393 catalog_manager.cc:5484] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135). New cstate: current_term: 1 leader_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:55.316319 31686 tablet_copy_source_session.cc:205] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:55.318642 4223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1022c4c6ee8642ca95250933a940fe7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:55.318481 2891 raft_consensus.cc:2932] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } } }
I20240502 21:26:55.321210 1833 raft_consensus.cc:2932] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } } }
W20240502 21:26:55.326370 2164 consensus_peers.cc:485] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d -> Peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Couldn't send request to peer a44f41b0b6644fb6aa29476aed8c964f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f13c3b9c4b7c471fa34156b781473ec4. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:55.326730 4249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b20df8896a2b423cb53c2c1a2a1b98e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:55.327409 31393 catalog_manager.cc:5484] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db reported cstate change: config changed from index -1 to 2, NON_VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) added. New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } } }
I20240502 21:26:55.332262 4216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182b3f15c06c4c07ade0a8854d3ecafd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:55.337481 31393 catalog_manager.cc:5484] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 9dec645b473f40578e726f3e50e219bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dec645b473f40578e726f3e50e219bf (127.24.226.134). New cstate: current_term: 1 leader_uuid: "9dec645b473f40578e726f3e50e219bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:55.341603 31387 catalog_manager.cc:5484] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d reported cstate change: term changed from 0 to 1, leader changed from <none> to d1406e92aec84ac585304e57bd37a40d (127.24.226.136). New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:55.346204 4249 tablet_copy_client.cc:783] T b20df8896a2b423cb53c2c1a2a1b98e2 P d1406e92aec84ac585304e57bd37a40d: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:55.346462 4249 tablet_copy_client.cc:647] T b20df8896a2b423cb53c2c1a2a1b98e2 P d1406e92aec84ac585304e57bd37a40d: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:55.350044 4249 tablet_copy_client.cc:523] T b20df8896a2b423cb53c2c1a2a1b98e2 P d1406e92aec84ac585304e57bd37a40d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:55.354532 4249 tablet_bootstrap.cc:492] T b20df8896a2b423cb53c2c1a2a1b98e2 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:55.358611 4275 tablet_copy_client.cc:308] T 577e7a586ad9459f8e77ce4b8f563c09 P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.133:39863
I20240502 21:26:55.359206 1854 tablet_copy_service.cc:140] P 1e04157b98dd4fad98858c290b73c3db: Received BeginTabletCopySession request for tablet 577e7a586ad9459f8e77ce4b8f563c09 from peer 05f4fac41c4f4959b9b316cb112d0ada ({username='slave'} at 127.24.226.130:43577)
I20240502 21:26:55.359334 1854 tablet_copy_service.cc:161] P 1e04157b98dd4fad98858c290b73c3db: Beginning new tablet copy session on tablet 577e7a586ad9459f8e77ce4b8f563c09 from peer 05f4fac41c4f4959b9b316cb112d0ada at {username='slave'} at 127.24.226.130:43577: session id = 05f4fac41c4f4959b9b316cb112d0ada-577e7a586ad9459f8e77ce4b8f563c09
I20240502 21:26:55.360093 4248 tablet_bootstrap.cc:492] T de2216a74f8344779a881651b66a85e5 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:55.360688 4248 tablet_bootstrap.cc:492] T de2216a74f8344779a881651b66a85e5 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap complete.
I20240502 21:26:55.361513 4248 ts_tablet_manager.cc:1367] T de2216a74f8344779a881651b66a85e5 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.099s user 0.019s sys 0.009s
I20240502 21:26:55.362182 1854 tablet_copy_source_session.cc:205] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:55.362540 4248 raft_consensus.cc:348] T de2216a74f8344779a881651b66a85e5 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } }
I20240502 21:26:55.362831 4248 raft_consensus.cc:729] T de2216a74f8344779a881651b66a85e5 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: LEARNER
I20240502 21:26:55.363085 4248 consensus_queue.cc:260] T de2216a74f8344779a881651b66a85e5 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } }
I20240502 21:26:55.364001 4277 raft_consensus.cc:482] T 43c02342d1ce4b53870e0495116ef173 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:55.364401 4277 raft_consensus.cc:504] T 43c02342d1ce4b53870e0495116ef173 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:55.365358 4248 ts_tablet_manager.cc:1398] T de2216a74f8344779a881651b66a85e5 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 21:26:55.365605 1965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43c02342d1ce4b53870e0495116ef173" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:26:55.365847 1965 raft_consensus.cc:2379] T 43c02342d1ce4b53870e0495116ef173 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1406e92aec84ac585304e57bd37a40d in current term 1: Already voted for candidate 1ade0354c44c4c37ae480c7a94b11e1b in this term.
I20240502 21:26:55.366052 31686 tablet_copy_service.cc:342] P 05f4fac41c4f4959b9b316cb112d0ada: Request end of tablet copy session 08d7a0680d244b55b93d17f9ab57b84c-de2216a74f8344779a881651b66a85e5 received from {username='slave'} at 127.24.226.131:52471
I20240502 21:26:55.366205 31686 tablet_copy_service.cc:433] P 05f4fac41c4f4959b9b316cb112d0ada: ending tablet copy session 08d7a0680d244b55b93d17f9ab57b84c-de2216a74f8344779a881651b66a85e5 on tablet de2216a74f8344779a881651b66a85e5 with peer 08d7a0680d244b55b93d17f9ab57b84c
I20240502 21:26:55.367211 4277 leader_election.cc:288] T 43c02342d1ce4b53870e0495116ef173 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:55.367364 2094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43c02342d1ce4b53870e0495116ef173" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:55.367940 4275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 577e7a586ad9459f8e77ce4b8f563c09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:55.367920 2163 leader_election.cc:302] T 43c02342d1ce4b53870e0495116ef173 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1406e92aec84ac585304e57bd37a40d; no voters: 1ade0354c44c4c37ae480c7a94b11e1b, 9dec645b473f40578e726f3e50e219bf
I20240502 21:26:55.369525 4277 raft_consensus.cc:3037] T 43c02342d1ce4b53870e0495116ef173 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:55.371248 1965 raft_consensus.cc:2932] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } } }
I20240502 21:26:55.372154 4235 tablet_copy_client.cc:308] T 46279270b2334633906af57c4b43e969 P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.135:38569
I20240502 21:26:55.372849 2118 tablet_copy_service.cc:140] P 1ade0354c44c4c37ae480c7a94b11e1b: Received BeginTabletCopySession request for tablet 46279270b2334633906af57c4b43e969 from peer 05f4fac41c4f4959b9b316cb112d0ada ({username='slave'} at 127.24.226.130:60987)
I20240502 21:26:55.373019 2118 tablet_copy_service.cc:161] P 1ade0354c44c4c37ae480c7a94b11e1b: Beginning new tablet copy session on tablet 46279270b2334633906af57c4b43e969 from peer 05f4fac41c4f4959b9b316cb112d0ada at {username='slave'} at 127.24.226.130:60987: session id = 05f4fac41c4f4959b9b316cb112d0ada-46279270b2334633906af57c4b43e969
I20240502 21:26:55.376643 2118 tablet_copy_source_session.cc:205] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:55.381814 4277 raft_consensus.cc:2726] T 43c02342d1ce4b53870e0495116ef173 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:26:55.381976 4235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46279270b2334633906af57c4b43e969. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:55.394620 4286 tablet_copy_client.cc:308] T 1ed225bdefc64ddd8eef2109bfd969df P d1406e92aec84ac585304e57bd37a40d: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.129:41039
I20240502 21:26:55.395601 31556 tablet_copy_service.cc:140] P a44f41b0b6644fb6aa29476aed8c964f: Received BeginTabletCopySession request for tablet 1ed225bdefc64ddd8eef2109bfd969df from peer d1406e92aec84ac585304e57bd37a40d ({username='slave'} at 127.24.226.136:52831)
I20240502 21:26:55.395759 31556 tablet_copy_service.cc:161] P a44f41b0b6644fb6aa29476aed8c964f: Beginning new tablet copy session on tablet 1ed225bdefc64ddd8eef2109bfd969df from peer d1406e92aec84ac585304e57bd37a40d at {username='slave'} at 127.24.226.136:52831: session id = d1406e92aec84ac585304e57bd37a40d-1ed225bdefc64ddd8eef2109bfd969df
I20240502 21:26:55.395947 4223 tablet_copy_client.cc:783] T 1022c4c6ee8642ca95250933a940fe7a P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:55.396149 4223 tablet_copy_client.cc:647] T 1022c4c6ee8642ca95250933a940fe7a P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:55.397264 4216 tablet_copy_client.cc:783] T 182b3f15c06c4c07ade0a8854d3ecafd P 9dec645b473f40578e726f3e50e219bf: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:55.398450 4216 tablet_copy_client.cc:647] T 182b3f15c06c4c07ade0a8854d3ecafd P 9dec645b473f40578e726f3e50e219bf: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:55.398604 4235 tablet_copy_client.cc:783] T 46279270b2334633906af57c4b43e969 P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:55.398698 31556 tablet_copy_source_session.cc:205] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:55.398875 4235 tablet_copy_client.cc:647] T 46279270b2334633906af57c4b43e969 P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:55.400156 4275 tablet_copy_client.cc:783] T 577e7a586ad9459f8e77ce4b8f563c09 P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:55.400357 4275 tablet_copy_client.cc:647] T 577e7a586ad9459f8e77ce4b8f563c09 P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:55.400635 4286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ed225bdefc64ddd8eef2109bfd969df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:55.402082 4275 tablet_copy_client.cc:523] T 577e7a586ad9459f8e77ce4b8f563c09 P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:55.403191 4223 tablet_copy_client.cc:523] T 1022c4c6ee8642ca95250933a940fe7a P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:55.407582 4286 tablet_copy_client.cc:783] T 1ed225bdefc64ddd8eef2109bfd969df P d1406e92aec84ac585304e57bd37a40d: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:55.407783 4286 tablet_copy_client.cc:647] T 1ed225bdefc64ddd8eef2109bfd969df P d1406e92aec84ac585304e57bd37a40d: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:55.407899 4235 tablet_copy_client.cc:523] T 46279270b2334633906af57c4b43e969 P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:55.408193 4216 tablet_copy_client.cc:523] T 182b3f15c06c4c07ade0a8854d3ecafd P 9dec645b473f40578e726f3e50e219bf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:55.410135 4235 tablet_bootstrap.cc:492] T 46279270b2334633906af57c4b43e969 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:55.412647 4223 tablet_bootstrap.cc:492] T 1022c4c6ee8642ca95250933a940fe7a P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:55.413656 4275 tablet_bootstrap.cc:492] T 577e7a586ad9459f8e77ce4b8f563c09 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:55.417233 4216 tablet_bootstrap.cc:492] T 182b3f15c06c4c07ade0a8854d3ecafd P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:55.418375 4286 tablet_copy_client.cc:523] T 1ed225bdefc64ddd8eef2109bfd969df P d1406e92aec84ac585304e57bd37a40d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:55.424530 4286 tablet_bootstrap.cc:492] T 1ed225bdefc64ddd8eef2109bfd969df P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:55.450017 4223 tablet_bootstrap.cc:492] T 1022c4c6ee8642ca95250933a940fe7a P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:55.450513 4223 tablet_bootstrap.cc:492] T 1022c4c6ee8642ca95250933a940fe7a P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap complete.
I20240502 21:26:55.450819 4223 ts_tablet_manager.cc:1367] T 1022c4c6ee8642ca95250933a940fe7a P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.038s user 0.009s sys 0.001s
I20240502 21:26:55.451682 4223 raft_consensus.cc:348] T 1022c4c6ee8642ca95250933a940fe7a P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } }
I20240502 21:26:55.453310 4223 raft_consensus.cc:729] T 1022c4c6ee8642ca95250933a940fe7a P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: LEARNER
I20240502 21:26:55.453629 4223 consensus_queue.cc:260] T 1022c4c6ee8642ca95250933a940fe7a P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } }
I20240502 21:26:55.454804 4223 ts_tablet_manager.cc:1398] T 1022c4c6ee8642ca95250933a940fe7a P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 21:26:55.456266 2251 tablet_copy_service.cc:342] P d1406e92aec84ac585304e57bd37a40d: Request end of tablet copy session 05f4fac41c4f4959b9b316cb112d0ada-1022c4c6ee8642ca95250933a940fe7a received from {username='slave'} at 127.24.226.130:56069
I20240502 21:26:55.456411 2251 tablet_copy_service.cc:433] P d1406e92aec84ac585304e57bd37a40d: ending tablet copy session 05f4fac41c4f4959b9b316cb112d0ada-1022c4c6ee8642ca95250933a940fe7a on tablet 1022c4c6ee8642ca95250933a940fe7a with peer 05f4fac41c4f4959b9b316cb112d0ada
I20240502 21:26:55.470963 31387 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:26:55.478725 4249 tablet_bootstrap.cc:492] T b20df8896a2b423cb53c2c1a2a1b98e2 P d1406e92aec84ac585304e57bd37a40d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:55.479157 4249 tablet_bootstrap.cc:492] T b20df8896a2b423cb53c2c1a2a1b98e2 P d1406e92aec84ac585304e57bd37a40d: Bootstrap complete.
I20240502 21:26:55.479455 4249 ts_tablet_manager.cc:1367] T b20df8896a2b423cb53c2c1a2a1b98e2 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.125s user 0.041s sys 0.008s
I20240502 21:26:55.480132 4249 raft_consensus.cc:348] T b20df8896a2b423cb53c2c1a2a1b98e2 P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } }
I20240502 21:26:55.480410 4249 raft_consensus.cc:729] T b20df8896a2b423cb53c2c1a2a1b98e2 P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: LEARNER
I20240502 21:26:55.480649 4249 consensus_queue.cc:260] T b20df8896a2b423cb53c2c1a2a1b98e2 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } }
I20240502 21:26:55.485014 4275 tablet_bootstrap.cc:492] T 577e7a586ad9459f8e77ce4b8f563c09 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:55.485666 4275 tablet_bootstrap.cc:492] T 577e7a586ad9459f8e77ce4b8f563c09 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap complete.
I20240502 21:26:55.485814 4235 tablet_bootstrap.cc:492] T 46279270b2334633906af57c4b43e969 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:55.485940 4275 ts_tablet_manager.cc:1367] T 577e7a586ad9459f8e77ce4b8f563c09 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.072s user 0.009s sys 0.013s
I20240502 21:26:55.486207 4235 tablet_bootstrap.cc:492] T 46279270b2334633906af57c4b43e969 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap complete.
I20240502 21:26:55.486450 4235 ts_tablet_manager.cc:1367] T 46279270b2334633906af57c4b43e969 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.076s user 0.005s sys 0.011s
I20240502 21:26:55.487121 4235 raft_consensus.cc:348] T 46279270b2334633906af57c4b43e969 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } }
I20240502 21:26:55.487349 4235 raft_consensus.cc:729] T 46279270b2334633906af57c4b43e969 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: LEARNER
I20240502 21:26:55.487059 4275 raft_consensus.cc:348] T 577e7a586ad9459f8e77ce4b8f563c09 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } }
I20240502 21:26:55.487504 4275 raft_consensus.cc:729] T 577e7a586ad9459f8e77ce4b8f563c09 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: LEARNER
I20240502 21:26:55.487535 4235 consensus_queue.cc:260] T 46279270b2334633906af57c4b43e969 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } }
I20240502 21:26:55.487788 4275 consensus_queue.cc:260] T 577e7a586ad9459f8e77ce4b8f563c09 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } }
I20240502 21:26:55.488343 4235 ts_tablet_manager.cc:1398] T 46279270b2334633906af57c4b43e969 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240502 21:26:55.489126 2118 tablet_copy_service.cc:342] P 1ade0354c44c4c37ae480c7a94b11e1b: Request end of tablet copy session 05f4fac41c4f4959b9b316cb112d0ada-46279270b2334633906af57c4b43e969 received from {username='slave'} at 127.24.226.130:60987
I20240502 21:26:55.489230 4249 ts_tablet_manager.cc:1398] T b20df8896a2b423cb53c2c1a2a1b98e2 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 21:26:55.489328 2118 tablet_copy_service.cc:433] P 1ade0354c44c4c37ae480c7a94b11e1b: ending tablet copy session 05f4fac41c4f4959b9b316cb112d0ada-46279270b2334633906af57c4b43e969 on tablet 46279270b2334633906af57c4b43e969 with peer 05f4fac41c4f4959b9b316cb112d0ada
I20240502 21:26:55.489554 4275 ts_tablet_manager.cc:1398] T 577e7a586ad9459f8e77ce4b8f563c09 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:26:55.490337 1854 tablet_copy_service.cc:342] P 1e04157b98dd4fad98858c290b73c3db: Request end of tablet copy session 05f4fac41c4f4959b9b316cb112d0ada-577e7a586ad9459f8e77ce4b8f563c09 received from {username='slave'} at 127.24.226.130:43577
I20240502 21:26:55.490478 1854 tablet_copy_service.cc:433] P 1e04157b98dd4fad98858c290b73c3db: ending tablet copy session 05f4fac41c4f4959b9b316cb112d0ada-577e7a586ad9459f8e77ce4b8f563c09 on tablet 577e7a586ad9459f8e77ce4b8f563c09 with peer 05f4fac41c4f4959b9b316cb112d0ada
I20240502 21:26:55.491304 31556 tablet_copy_service.cc:342] P a44f41b0b6644fb6aa29476aed8c964f: Request end of tablet copy session d1406e92aec84ac585304e57bd37a40d-b20df8896a2b423cb53c2c1a2a1b98e2 received from {username='slave'} at 127.24.226.136:52831
I20240502 21:26:55.491454 31556 tablet_copy_service.cc:433] P a44f41b0b6644fb6aa29476aed8c964f: ending tablet copy session d1406e92aec84ac585304e57bd37a40d-b20df8896a2b423cb53c2c1a2a1b98e2 on tablet b20df8896a2b423cb53c2c1a2a1b98e2 with peer d1406e92aec84ac585304e57bd37a40d
I20240502 21:26:55.505030 2225 raft_consensus.cc:1261] T 43c02342d1ce4b53870e0495116ef173 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 1ade0354c44c4c37ae480c7a94b11e1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:55.505551 1964 raft_consensus.cc:1261] T 43c02342d1ce4b53870e0495116ef173 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer 1ade0354c44c4c37ae480c7a94b11e1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:55.506278 4268 consensus_queue.cc:1035] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:55.506680 4268 consensus_queue.cc:1035] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:55.525561 4309 ts_tablet_manager.cc:902] T 83904952a5a541ecac508449e23e70dd P 1e04157b98dd4fad98858c290b73c3db: Initiating tablet copy from peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:55.533169 31667 raft_consensus.cc:1203] T 1022c4c6ee8642ca95250933a940fe7a P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:26:55.544684 31387 catalog_manager.cc:5576] TS d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907) does not have the latest schema for tablet 43c02342d1ce4b53870e0495116ef173 (table rebalancer_dynamic_table [id=03638b9bda5347dcbf84b97a9589a9c6]). Expected version 1 got 0
I20240502 21:26:55.551234 31393 catalog_manager.cc:5576] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753) does not have the latest schema for tablet 43c02342d1ce4b53870e0495116ef173 (table rebalancer_dynamic_table [id=03638b9bda5347dcbf84b97a9589a9c6]). Expected version 1 got 0
I20240502 21:26:55.570266 3818 consensus_queue.cc:1035] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:55.585347 4320 tablet.cc:1722] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:55.585945 4320 tablet.cc:1620] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b: MemRowSet was empty: no flush needed.
I20240502 21:26:55.584777 4321 tablet.cc:1722] T 43c02342d1ce4b53870e0495116ef173 P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:55.587015 4321 tablet.cc:1620] T 43c02342d1ce4b53870e0495116ef173 P 9dec645b473f40578e726f3e50e219bf: MemRowSet was empty: no flush needed.
I20240502 21:26:55.588343 31387 catalog_manager.cc:5576] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753) does not have the latest schema for tablet 43c02342d1ce4b53870e0495116ef173 (table rebalancer_dynamic_table [id=03638b9bda5347dcbf84b97a9589a9c6]). Expected version 1 got 0
I20240502 21:26:55.600903 31393 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=03638b9bda5347dcbf84b97a9589a9c6] alter complete (version 1)
I20240502 21:26:55.601248 1965 raft_consensus.cc:1203] T 0b8d119ee8474a40ac08df61df5b8423 P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:26:55.611411 4309 tablet_copy_client.cc:308] T 83904952a5a541ecac508449e23e70dd P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.132:35933
I20240502 21:26:55.612177 31947 tablet_copy_service.cc:140] P 2d50f33557944bb480ea32199d9ee4f1: Received BeginTabletCopySession request for tablet 83904952a5a541ecac508449e23e70dd from peer 1e04157b98dd4fad98858c290b73c3db ({username='slave'} at 127.24.226.133:55617)
I20240502 21:26:55.612413 31947 tablet_copy_service.cc:161] P 2d50f33557944bb480ea32199d9ee4f1: Beginning new tablet copy session on tablet 83904952a5a541ecac508449e23e70dd from peer 1e04157b98dd4fad98858c290b73c3db at {username='slave'} at 127.24.226.133:55617: session id = 1e04157b98dd4fad98858c290b73c3db-83904952a5a541ecac508449e23e70dd
I20240502 21:26:55.613673 2230 raft_consensus.cc:1203] T b20df8896a2b423cb53c2c1a2a1b98e2 P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:26:55.615363 31947 tablet_copy_source_session.cc:205] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:55.619467 4309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83904952a5a541ecac508449e23e70dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:55.627192 4309 tablet_copy_client.cc:783] T 83904952a5a541ecac508449e23e70dd P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:55.627503 4309 tablet_copy_client.cc:647] T 83904952a5a541ecac508449e23e70dd P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:55.629424 4309 tablet_copy_client.cc:523] T 83904952a5a541ecac508449e23e70dd P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:55.629853 4265 consensus_queue.cc:1035] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:55.632764 4309 tablet_bootstrap.cc:492] T 83904952a5a541ecac508449e23e70dd P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:55.635540 2891 consensus_queue.cc:1035] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:55.638020 4323 tablet.cc:1722] T 43c02342d1ce4b53870e0495116ef173 P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:55.638864 4323 tablet.cc:1620] T 43c02342d1ce4b53870e0495116ef173 P d1406e92aec84ac585304e57bd37a40d: MemRowSet was empty: no flush needed.
I20240502 21:26:55.641896 31394 catalog_manager.cc:5576] TS d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907) does not have the latest schema for tablet 43c02342d1ce4b53870e0495116ef173 (table rebalancer_dynamic_table [id=03638b9bda5347dcbf84b97a9589a9c6]). Expected version 1 got 0
I20240502 21:26:55.666111 4216 tablet_bootstrap.cc:492] T 182b3f15c06c4c07ade0a8854d3ecafd P 9dec645b473f40578e726f3e50e219bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:55.666543 4216 tablet_bootstrap.cc:492] T 182b3f15c06c4c07ade0a8854d3ecafd P 9dec645b473f40578e726f3e50e219bf: Bootstrap complete.
I20240502 21:26:55.666785 4216 ts_tablet_manager.cc:1367] T 182b3f15c06c4c07ade0a8854d3ecafd P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.250s user 0.055s sys 0.001s
I20240502 21:26:55.667855 4216 raft_consensus.cc:348] T 182b3f15c06c4c07ade0a8854d3ecafd P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:26:55.668133 4216 raft_consensus.cc:729] T 182b3f15c06c4c07ade0a8854d3ecafd P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: LEARNER
I20240502 21:26:55.668363 4216 consensus_queue.cc:260] T 182b3f15c06c4c07ade0a8854d3ecafd P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:26:55.668965 4216 ts_tablet_manager.cc:1398] T 182b3f15c06c4c07ade0a8854d3ecafd P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 21:26:55.671231 31686 tablet_copy_service.cc:342] P 05f4fac41c4f4959b9b316cb112d0ada: Request end of tablet copy session 9dec645b473f40578e726f3e50e219bf-182b3f15c06c4c07ade0a8854d3ecafd received from {username='slave'} at 127.24.226.134:40513
I20240502 21:26:55.671435 31686 tablet_copy_service.cc:433] P 05f4fac41c4f4959b9b316cb112d0ada: ending tablet copy session 9dec645b473f40578e726f3e50e219bf-182b3f15c06c4c07ade0a8854d3ecafd on tablet 182b3f15c06c4c07ade0a8854d3ecafd with peer 9dec645b473f40578e726f3e50e219bf
I20240502 21:26:55.672487 3765 consensus_queue.cc:1035] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:55.673767 4167 consensus_queue.cc:1035] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:55.683214 31795 raft_consensus.cc:1203] T de2216a74f8344779a881651b66a85e5 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:26:55.689183 31667 raft_consensus.cc:1203] T 577e7a586ad9459f8e77ce4b8f563c09 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:26:55.694976 3946 consensus_queue.cc:1035] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:55.732899 31393 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:26:55.748117 4320 tablet.cc:1722] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:55.748591 4320 tablet.cc:1620] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b: MemRowSet was empty: no flush needed.
I20240502 21:26:55.749022 4321 tablet.cc:1722] T 43c02342d1ce4b53870e0495116ef173 P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:55.749451 4321 tablet.cc:1620] T 43c02342d1ce4b53870e0495116ef173 P 9dec645b473f40578e726f3e50e219bf: MemRowSet was empty: no flush needed.
I20240502 21:26:55.751833 4323 tablet.cc:1722] T 43c02342d1ce4b53870e0495116ef173 P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:55.752244 4323 tablet.cc:1620] T 43c02342d1ce4b53870e0495116ef173 P d1406e92aec84ac585304e57bd37a40d: MemRowSet was empty: no flush needed.
I20240502 21:26:55.754575 4286 tablet_bootstrap.cc:492] T 1ed225bdefc64ddd8eef2109bfd969df P d1406e92aec84ac585304e57bd37a40d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:55.755157 4286 tablet_bootstrap.cc:492] T 1ed225bdefc64ddd8eef2109bfd969df P d1406e92aec84ac585304e57bd37a40d: Bootstrap complete.
I20240502 21:26:55.755456 4286 ts_tablet_manager.cc:1367] T 1ed225bdefc64ddd8eef2109bfd969df P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.331s user 0.059s sys 0.029s
I20240502 21:26:55.756255 4286 raft_consensus.cc:348] T 1ed225bdefc64ddd8eef2109bfd969df P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } }
I20240502 21:26:55.756539 4286 raft_consensus.cc:729] T 1ed225bdefc64ddd8eef2109bfd969df P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: LEARNER
I20240502 21:26:55.758198 4286 consensus_queue.cc:260] T 1ed225bdefc64ddd8eef2109bfd969df P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } }
I20240502 21:26:55.758847 4286 ts_tablet_manager.cc:1398] T 1ed225bdefc64ddd8eef2109bfd969df P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:55.763890 4309 tablet_bootstrap.cc:492] T 83904952a5a541ecac508449e23e70dd P 1e04157b98dd4fad98858c290b73c3db: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:55.764335 4309 tablet_bootstrap.cc:492] T 83904952a5a541ecac508449e23e70dd P 1e04157b98dd4fad98858c290b73c3db: Bootstrap complete.
I20240502 21:26:55.764654 4309 ts_tablet_manager.cc:1367] T 83904952a5a541ecac508449e23e70dd P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.132s user 0.089s sys 0.016s
I20240502 21:26:55.765475 4309 raft_consensus.cc:348] T 83904952a5a541ecac508449e23e70dd P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } }
I20240502 21:26:55.765779 4309 raft_consensus.cc:729] T 83904952a5a541ecac508449e23e70dd P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: LEARNER
I20240502 21:26:55.765973 4309 consensus_queue.cc:260] T 83904952a5a541ecac508449e23e70dd P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } }
I20240502 21:26:55.766536 4309 ts_tablet_manager.cc:1398] T 83904952a5a541ecac508449e23e70dd P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:26:55.768711 31388 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=03638b9bda5347dcbf84b97a9589a9c6] alter complete (version 2)
I20240502 21:26:55.769058 2230 raft_consensus.cc:1203] T 1ed225bdefc64ddd8eef2109bfd969df P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:26:55.770560 31556 tablet_copy_service.cc:342] P a44f41b0b6644fb6aa29476aed8c964f: Request end of tablet copy session d1406e92aec84ac585304e57bd37a40d-1ed225bdefc64ddd8eef2109bfd969df received from {username='slave'} at 127.24.226.136:52831
I20240502 21:26:55.770677 31556 tablet_copy_service.cc:433] P a44f41b0b6644fb6aa29476aed8c964f: ending tablet copy session d1406e92aec84ac585304e57bd37a40d-1ed225bdefc64ddd8eef2109bfd969df on tablet 1ed225bdefc64ddd8eef2109bfd969df with peer d1406e92aec84ac585304e57bd37a40d
I20240502 21:26:55.786157 4341 ts_tablet_manager.cc:902] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f: Initiating tablet copy from peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
W20240502 21:26:55.786741 1770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.226.130:37645, user_credentials={real_user=slave}} blocked reactor thread for 96936.1us
I20240502 21:26:55.787482 31947 tablet_copy_service.cc:342] P 2d50f33557944bb480ea32199d9ee4f1: Request end of tablet copy session 1e04157b98dd4fad98858c290b73c3db-83904952a5a541ecac508449e23e70dd received from {username='slave'} at 127.24.226.133:55617
I20240502 21:26:55.787602 31947 tablet_copy_service.cc:433] P 2d50f33557944bb480ea32199d9ee4f1: ending tablet copy session 1e04157b98dd4fad98858c290b73c3db-83904952a5a541ecac508449e23e70dd on tablet 83904952a5a541ecac508449e23e70dd with peer 1e04157b98dd4fad98858c290b73c3db
I20240502 21:26:55.789872 1964 raft_consensus.cc:1203] T 182b3f15c06c4c07ade0a8854d3ecafd P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:26:55.813248 4341 tablet_copy_client.cc:308] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.136:34907
I20240502 21:26:55.814255 2251 tablet_copy_service.cc:140] P d1406e92aec84ac585304e57bd37a40d: Received BeginTabletCopySession request for tablet f13c3b9c4b7c471fa34156b781473ec4 from peer a44f41b0b6644fb6aa29476aed8c964f ({username='slave'} at 127.24.226.129:40523)
I20240502 21:26:55.814414 2251 tablet_copy_service.cc:161] P d1406e92aec84ac585304e57bd37a40d: Beginning new tablet copy session on tablet f13c3b9c4b7c471fa34156b781473ec4 from peer a44f41b0b6644fb6aa29476aed8c964f at {username='slave'} at 127.24.226.129:40523: session id = a44f41b0b6644fb6aa29476aed8c964f-f13c3b9c4b7c471fa34156b781473ec4
I20240502 21:26:55.817126 2251 tablet_copy_source_session.cc:205] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:55.821475 4341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f13c3b9c4b7c471fa34156b781473ec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:55.828722 4341 tablet_copy_client.cc:783] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:55.829015 4341 tablet_copy_client.cc:647] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:55.831209 4341 tablet_copy_client.cc:523] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:55.837450 4341 tablet_bootstrap.cc:492] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:55.856309 4341 tablet_bootstrap.cc:492] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:55.856767 4341 tablet_bootstrap.cc:492] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap complete.
I20240502 21:26:55.857019 4341 ts_tablet_manager.cc:1367] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.000s
I20240502 21:26:55.858588 4341 raft_consensus.cc:348] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } }
I20240502 21:26:55.858655 31393 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:55.858942 4341 raft_consensus.cc:729] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: LEARNER
I20240502 21:26:55.859244 4341 consensus_queue.cc:260] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } }
I20240502 21:26:55.859908 4341 ts_tablet_manager.cc:1398] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 21:26:55.864648 2251 tablet_copy_service.cc:342] P d1406e92aec84ac585304e57bd37a40d: Request end of tablet copy session a44f41b0b6644fb6aa29476aed8c964f-f13c3b9c4b7c471fa34156b781473ec4 received from {username='slave'} at 127.24.226.129:40523
I20240502 21:26:55.864805 2251 tablet_copy_service.cc:433] P d1406e92aec84ac585304e57bd37a40d: ending tablet copy session a44f41b0b6644fb6aa29476aed8c964f-f13c3b9c4b7c471fa34156b781473ec4 on tablet f13c3b9c4b7c471fa34156b781473ec4 with peer a44f41b0b6644fb6aa29476aed8c964f
I20240502 21:26:55.875728 31645 tablet_service.cc:1467] Processing CreateTablet for tablet 1bf575f91df54d75a9ac519924fb62e7 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=1be6e16d4c9b4aaabceb108765a77557]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:55.876185 31645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bf575f91df54d75a9ac519924fb62e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:55.884712 2208 tablet_service.cc:1467] Processing CreateTablet for tablet 1bf575f91df54d75a9ac519924fb62e7 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=1be6e16d4c9b4aaabceb108765a77557]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:55.885157 2208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bf575f91df54d75a9ac519924fb62e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:55.884313 31905 tablet_service.cc:1467] Processing CreateTablet for tablet 1bf575f91df54d75a9ac519924fb62e7 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=1be6e16d4c9b4aaabceb108765a77557]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:55.886641 31905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bf575f91df54d75a9ac519924fb62e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:55.887970 1946 tablet_service.cc:1467] Processing CreateTablet for tablet b1d59ebecac34cf7b33424cc73cff030 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=1be6e16d4c9b4aaabceb108765a77557]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:55.888492 1946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1d59ebecac34cf7b33424cc73cff030. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:55.888650 31393 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:26:55.891542 31647 tablet_service.cc:1467] Processing CreateTablet for tablet 9c40a6852649404f8928c2642e8ac3be (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=1be6e16d4c9b4aaabceb108765a77557]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:55.891952 31647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c40a6852649404f8928c2642e8ac3be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:55.893898 2076 tablet_service.cc:1467] Processing CreateTablet for tablet b1d59ebecac34cf7b33424cc73cff030 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=1be6e16d4c9b4aaabceb108765a77557]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:55.894438 2076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1d59ebecac34cf7b33424cc73cff030. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:55.898907 2210 tablet_service.cc:1467] Processing CreateTablet for tablet b1d59ebecac34cf7b33424cc73cff030 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=1be6e16d4c9b4aaabceb108765a77557]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:55.899050 2209 tablet_service.cc:1467] Processing CreateTablet for tablet 9c40a6852649404f8928c2642e8ac3be (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=1be6e16d4c9b4aaabceb108765a77557]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:55.899402 2210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1d59ebecac34cf7b33424cc73cff030. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:55.905355 2077 tablet_service.cc:1467] Processing CreateTablet for tablet 9c40a6852649404f8928c2642e8ac3be (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=1be6e16d4c9b4aaabceb108765a77557]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:55.905686 2077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c40a6852649404f8928c2642e8ac3be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:55.905838 2209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c40a6852649404f8928c2642e8ac3be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:55.915532 4320 tablet.cc:1722] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:55.917402 4323 tablet.cc:1722] T 43c02342d1ce4b53870e0495116ef173 P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:55.919211 31667 raft_consensus.cc:1203] T 46279270b2334633906af57c4b43e969 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:26:55.920517 4321 tablet.cc:1722] T 43c02342d1ce4b53870e0495116ef173 P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:55.926556 4363 tablet_bootstrap.cc:492] T 1bf575f91df54d75a9ac519924fb62e7 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:55.929153 31387 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table_new_name [id=03638b9bda5347dcbf84b97a9589a9c6] alter complete (version 3)
I20240502 21:26:55.933966 4363 tablet_bootstrap.cc:654] T 1bf575f91df54d75a9ac519924fb62e7 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:55.942644 4363 tablet_bootstrap.cc:492] T 1bf575f91df54d75a9ac519924fb62e7 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:55.945783 4363 ts_tablet_manager.cc:1367] T 1bf575f91df54d75a9ac519924fb62e7 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.004s
I20240502 21:26:55.945958 4365 tablet_bootstrap.cc:492] T b1d59ebecac34cf7b33424cc73cff030 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:55.946930 4363 raft_consensus.cc:348] T 1bf575f91df54d75a9ac519924fb62e7 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:55.947683 4365 tablet_bootstrap.cc:654] T b1d59ebecac34cf7b33424cc73cff030 P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:55.949133 4367 tablet_bootstrap.cc:492] T b1d59ebecac34cf7b33424cc73cff030 P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:55.950670 4363 raft_consensus.cc:374] T 1bf575f91df54d75a9ac519924fb62e7 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:55.950775 4363 raft_consensus.cc:729] T 1bf575f91df54d75a9ac519924fb62e7 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:55.950989 4363 consensus_queue.cc:260] T 1bf575f91df54d75a9ac519924fb62e7 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:55.951589 4363 ts_tablet_manager.cc:1398] T 1bf575f91df54d75a9ac519924fb62e7 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240502 21:26:55.954084 4363 tablet_bootstrap.cc:492] T 9c40a6852649404f8928c2642e8ac3be P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:55.955729 4363 tablet_bootstrap.cc:654] T 9c40a6852649404f8928c2642e8ac3be P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:55.960934 4364 tablet_bootstrap.cc:492] T 1bf575f91df54d75a9ac519924fb62e7 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:55.962963 4364 tablet_bootstrap.cc:654] T 1bf575f91df54d75a9ac519924fb62e7 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:55.964506 4367 tablet_bootstrap.cc:654] T b1d59ebecac34cf7b33424cc73cff030 P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:55.966761 4367 tablet_bootstrap.cc:492] T b1d59ebecac34cf7b33424cc73cff030 P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:26:55.966944 4367 ts_tablet_manager.cc:1367] T b1d59ebecac34cf7b33424cc73cff030 P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240502 21:26:55.968057 4367 raft_consensus.cc:348] T b1d59ebecac34cf7b33424cc73cff030 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:55.968797 4367 raft_consensus.cc:374] T b1d59ebecac34cf7b33424cc73cff030 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:55.969602 4367 raft_consensus.cc:729] T b1d59ebecac34cf7b33424cc73cff030 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:26:55.969790 4365 tablet_bootstrap.cc:492] T b1d59ebecac34cf7b33424cc73cff030 P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:55.969938 4365 ts_tablet_manager.cc:1367] T b1d59ebecac34cf7b33424cc73cff030 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20240502 21:26:55.970633 4367 consensus_queue.cc:260] T b1d59ebecac34cf7b33424cc73cff030 P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:55.971875 4365 raft_consensus.cc:348] T b1d59ebecac34cf7b33424cc73cff030 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:55.972116 4365 raft_consensus.cc:374] T b1d59ebecac34cf7b33424cc73cff030 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:55.972222 4365 raft_consensus.cc:729] T b1d59ebecac34cf7b33424cc73cff030 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:55.972465 4365 consensus_queue.cc:260] T b1d59ebecac34cf7b33424cc73cff030 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:55.973309 4363 tablet_bootstrap.cc:492] T 9c40a6852649404f8928c2642e8ac3be P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:55.973485 4363 ts_tablet_manager.cc:1367] T 9c40a6852649404f8928c2642e8ac3be P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.002s
I20240502 21:26:55.974046 4365 ts_tablet_manager.cc:1398] T b1d59ebecac34cf7b33424cc73cff030 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 21:26:55.974316 4365 tablet_bootstrap.cc:492] T 9c40a6852649404f8928c2642e8ac3be P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:55.974215 4363 raft_consensus.cc:348] T 9c40a6852649404f8928c2642e8ac3be P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:55.974479 4363 raft_consensus.cc:374] T 9c40a6852649404f8928c2642e8ac3be P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:55.974607 4363 raft_consensus.cc:729] T 9c40a6852649404f8928c2642e8ac3be P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:55.975358 4363 consensus_queue.cc:260] T 9c40a6852649404f8928c2642e8ac3be P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:55.975872 4365 tablet_bootstrap.cc:654] T 9c40a6852649404f8928c2642e8ac3be P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:55.976979 4366 tablet_bootstrap.cc:492] T 1bf575f91df54d75a9ac519924fb62e7 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:55.977321 4363 ts_tablet_manager.cc:1398] T 9c40a6852649404f8928c2642e8ac3be P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:26:55.977896 4367 ts_tablet_manager.cc:1398] T b1d59ebecac34cf7b33424cc73cff030 P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 21:26:55.981431 4364 tablet_bootstrap.cc:492] T 1bf575f91df54d75a9ac519924fb62e7 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:55.981590 4271 raft_consensus.cc:482] T b1d59ebecac34cf7b33424cc73cff030 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:55.981734 4271 raft_consensus.cc:504] T b1d59ebecac34cf7b33424cc73cff030 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:55.982400 4271 leader_election.cc:288] T b1d59ebecac34cf7b33424cc73cff030 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:55.983031 2225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1d59ebecac34cf7b33424cc73cff030" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:26:55.983160 4366 tablet_bootstrap.cc:654] T 1bf575f91df54d75a9ac519924fb62e7 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:55.984040 4364 ts_tablet_manager.cc:1367] T 1bf575f91df54d75a9ac519924fb62e7 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
W20240502 21:26:55.984038 2032 leader_election.cc:341] T b1d59ebecac34cf7b33424cc73cff030 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:55.984151 1964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1d59ebecac34cf7b33424cc73cff030" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:26:55.984351 1964 raft_consensus.cc:2454] T b1d59ebecac34cf7b33424cc73cff030 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 0.
I20240502 21:26:55.984807 2030 leader_election.cc:302] T b1d59ebecac34cf7b33424cc73cff030 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 9dec645b473f40578e726f3e50e219bf; no voters: d1406e92aec84ac585304e57bd37a40d
I20240502 21:26:55.985220 4271 raft_consensus.cc:2781] T b1d59ebecac34cf7b33424cc73cff030 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:55.985317 4271 raft_consensus.cc:482] T b1d59ebecac34cf7b33424cc73cff030 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:55.985379 4271 raft_consensus.cc:3037] T b1d59ebecac34cf7b33424cc73cff030 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:55.987674 4271 raft_consensus.cc:504] T b1d59ebecac34cf7b33424cc73cff030 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:55.988863 1964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1d59ebecac34cf7b33424cc73cff030" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf"
I20240502 21:26:55.989027 1964 raft_consensus.cc:3037] T b1d59ebecac34cf7b33424cc73cff030 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:55.991529 1964 raft_consensus.cc:2454] T b1d59ebecac34cf7b33424cc73cff030 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 1.
I20240502 21:26:55.992921 2030 leader_election.cc:302] T b1d59ebecac34cf7b33424cc73cff030 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:26:55.993216 4268 raft_consensus.cc:2781] T b1d59ebecac34cf7b33424cc73cff030 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:55.993335 4268 raft_consensus.cc:686] T b1d59ebecac34cf7b33424cc73cff030 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEADER]: Becoming Leader. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Running, Role: LEADER
I20240502 21:26:55.993631 4268 consensus_queue.cc:237] T b1d59ebecac34cf7b33424cc73cff030 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:55.995163 4271 leader_election.cc:288] T b1d59ebecac34cf7b33424cc73cff030 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 election: Requested vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:55.995977 1833 raft_consensus.cc:1203] T 83904952a5a541ecac508449e23e70dd P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:26:56.000164 4364 raft_consensus.cc:348] T 1bf575f91df54d75a9ac519924fb62e7 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:56.000456 4364 raft_consensus.cc:374] T 1bf575f91df54d75a9ac519924fb62e7 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:56.001495 2230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1d59ebecac34cf7b33424cc73cff030" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:26:56.002148 4364 raft_consensus.cc:729] T 1bf575f91df54d75a9ac519924fb62e7 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
W20240502 21:26:56.002213 2032 leader_election.cc:341] T b1d59ebecac34cf7b33424cc73cff030 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:56.003028 4364 consensus_queue.cc:260] T 1bf575f91df54d75a9ac519924fb62e7 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:56.005714 4364 ts_tablet_manager.cc:1398] T 1bf575f91df54d75a9ac519924fb62e7 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.022s user 0.000s sys 0.002s
I20240502 21:26:56.010035 4366 tablet_bootstrap.cc:492] T 1bf575f91df54d75a9ac519924fb62e7 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:56.010835 4365 tablet_bootstrap.cc:492] T 9c40a6852649404f8928c2642e8ac3be P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:56.011001 4365 ts_tablet_manager.cc:1367] T 9c40a6852649404f8928c2642e8ac3be P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.037s user 0.000s sys 0.003s
I20240502 21:26:56.011029 4366 ts_tablet_manager.cc:1367] T 1bf575f91df54d75a9ac519924fb62e7 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.000s
I20240502 21:26:56.012178 4366 raft_consensus.cc:348] T 1bf575f91df54d75a9ac519924fb62e7 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:56.013865 4365 raft_consensus.cc:348] T 9c40a6852649404f8928c2642e8ac3be P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:56.014200 4365 raft_consensus.cc:374] T 9c40a6852649404f8928c2642e8ac3be P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:56.014353 4365 raft_consensus.cc:729] T 9c40a6852649404f8928c2642e8ac3be P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:56.015036 4366 raft_consensus.cc:374] T 1bf575f91df54d75a9ac519924fb62e7 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:56.015112 4365 consensus_queue.cc:260] T 9c40a6852649404f8928c2642e8ac3be P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:56.015738 4365 ts_tablet_manager.cc:1398] T 9c40a6852649404f8928c2642e8ac3be P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 21:26:56.031235 4366 raft_consensus.cc:729] T 1bf575f91df54d75a9ac519924fb62e7 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:56.031710 4366 consensus_queue.cc:260] T 1bf575f91df54d75a9ac519924fb62e7 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:56.032382 31387 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:56.032677 31387 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:56.036927 31387 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 43c02342d1ce4b53870e0495116ef173
I20240502 21:26:56.037851 2076 tablet_service.cc:1514] Processing DeleteTablet for tablet 43c02342d1ce4b53870e0495116ef173 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:56 UTC) from {username='slave'} at 127.0.0.1:36740
I20240502 21:26:56.037848 2208 tablet_service.cc:1514] Processing DeleteTablet for tablet 43c02342d1ce4b53870e0495116ef173 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:56 UTC) from {username='slave'} at 127.0.0.1:36984
I20240502 21:26:56.038261 1946 tablet_service.cc:1514] Processing DeleteTablet for tablet 43c02342d1ce4b53870e0495116ef173 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:56 UTC) from {username='slave'} at 127.0.0.1:33002
I20240502 21:26:56.039047 4324 raft_consensus.cc:1050] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada: attempting to promote NON_VOTER 9dec645b473f40578e726f3e50e219bf to VOTER
I20240502 21:26:56.039820 4324 consensus_queue.cc:237] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:26:56.056248 31387 catalog_manager.cc:5484] T b1d59ebecac34cf7b33424cc73cff030 P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135). New cstate: current_term: 1 leader_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:56.061219 4387 tablet_replica.cc:330] T 43c02342d1ce4b53870e0495116ef173 P d1406e92aec84ac585304e57bd37a40d: stopping tablet replica
I20240502 21:26:56.061631 4387 raft_consensus.cc:2229] T 43c02342d1ce4b53870e0495116ef173 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:56.062153 4387 raft_consensus.cc:2258] T 43c02342d1ce4b53870e0495116ef173 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:56.063320 4366 ts_tablet_manager.cc:1398] T 1bf575f91df54d75a9ac519924fb62e7 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.052s user 0.002s sys 0.000s
I20240502 21:26:56.063797 4387 ts_tablet_manager.cc:1875] T 43c02342d1ce4b53870e0495116ef173 P d1406e92aec84ac585304e57bd37a40d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:56.064911 4366 tablet_bootstrap.cc:492] T 9c40a6852649404f8928c2642e8ac3be P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:56.066989 4366 tablet_bootstrap.cc:654] T 9c40a6852649404f8928c2642e8ac3be P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:56.083513 4387 ts_tablet_manager.cc:1888] T 43c02342d1ce4b53870e0495116ef173 P d1406e92aec84ac585304e57bd37a40d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:56.085198 4387 log.cc:1196] T 43c02342d1ce4b53870e0495116ef173 P d1406e92aec84ac585304e57bd37a40d: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal/wals/43c02342d1ce4b53870e0495116ef173
I20240502 21:26:56.085994 4387 ts_tablet_manager.cc:1909] T 43c02342d1ce4b53870e0495116ef173 P d1406e92aec84ac585304e57bd37a40d: Deleting consensus metadata
I20240502 21:26:56.087242 4388 tablet_replica.cc:330] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b: stopping tablet replica
I20240502 21:26:56.087512 4388 raft_consensus.cc:2229] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:56.087513 31381 catalog_manager.cc:4830] TS d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): tablet 43c02342d1ce4b53870e0495116ef173 (table rebalancer_dynamic_table_new_name [id=03638b9bda5347dcbf84b97a9589a9c6]) successfully deleted
I20240502 21:26:56.090742 4243 raft_consensus.cc:1050] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d: attempting to promote NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada to VOTER
I20240502 21:26:56.091495 4243 consensus_queue.cc:237] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } }
I20240502 21:26:56.103722 4389 tablet_replica.cc:330] T 43c02342d1ce4b53870e0495116ef173 P 9dec645b473f40578e726f3e50e219bf: stopping tablet replica
I20240502 21:26:56.104055 4389 raft_consensus.cc:2229] T 43c02342d1ce4b53870e0495116ef173 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:56.104809 4389 raft_consensus.cc:2258] T 43c02342d1ce4b53870e0495116ef173 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:56.106076 4389 ts_tablet_manager.cc:1875] T 43c02342d1ce4b53870e0495116ef173 P 9dec645b473f40578e726f3e50e219bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:56.110680 4388 raft_consensus.cc:2258] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:56.112028 4366 tablet_bootstrap.cc:492] T 9c40a6852649404f8928c2642e8ac3be P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:56.112275 4366 ts_tablet_manager.cc:1367] T 9c40a6852649404f8928c2642e8ac3be P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.000s
I20240502 21:26:56.112557 4268 raft_consensus.cc:482] T 9c40a6852649404f8928c2642e8ac3be P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:56.112715 4268 raft_consensus.cc:504] T 9c40a6852649404f8928c2642e8ac3be P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:56.113308 4268 leader_election.cc:288] T 9c40a6852649404f8928c2642e8ac3be P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:56.113191 4366 raft_consensus.cc:348] T 9c40a6852649404f8928c2642e8ac3be P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:56.113487 4366 raft_consensus.cc:374] T 9c40a6852649404f8928c2642e8ac3be P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:56.113596 4366 raft_consensus.cc:729] T 9c40a6852649404f8928c2642e8ac3be P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:56.113842 4366 consensus_queue.cc:260] T 9c40a6852649404f8928c2642e8ac3be P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:56.114537 4388 ts_tablet_manager.cc:1875] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:56.115821 31667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c40a6852649404f8928c2642e8ac3be" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:56.116046 31667 raft_consensus.cc:2454] T 9c40a6852649404f8928c2642e8ac3be P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 0.
I20240502 21:26:56.116478 2033 leader_election.cc:302] T 9c40a6852649404f8928c2642e8ac3be P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 1ade0354c44c4c37ae480c7a94b11e1b; no voters:
I20240502 21:26:56.116767 4268 raft_consensus.cc:2781] T 9c40a6852649404f8928c2642e8ac3be P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:56.116853 4268 raft_consensus.cc:482] T 9c40a6852649404f8928c2642e8ac3be P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:56.116961 4268 raft_consensus.cc:3037] T 9c40a6852649404f8928c2642e8ac3be P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:56.118470 1965 raft_consensus.cc:1261] T 0b8d119ee8474a40ac08df61df5b8423 P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:56.119333 4268 raft_consensus.cc:504] T 9c40a6852649404f8928c2642e8ac3be P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:56.119834 4268 leader_election.cc:288] T 9c40a6852649404f8928c2642e8ac3be P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 election: Requested vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:56.120368 3859 consensus_queue.cc:1035] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:56.120930 31667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c40a6852649404f8928c2642e8ac3be" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:56.121114 31667 raft_consensus.cc:3037] T 9c40a6852649404f8928c2642e8ac3be P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:56.121261 3859 raft_consensus.cc:1050] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada: attempting to promote NON_VOTER 08d7a0680d244b55b93d17f9ab57b84c to VOTER
I20240502 21:26:56.122069 3859 consensus_queue.cc:237] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } }
I20240502 21:26:56.123584 31667 raft_consensus.cc:2454] T 9c40a6852649404f8928c2642e8ac3be P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 1.
I20240502 21:26:56.124348 2033 leader_election.cc:302] T 9c40a6852649404f8928c2642e8ac3be P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 1ade0354c44c4c37ae480c7a94b11e1b; no voters:
I20240502 21:26:56.124897 4271 raft_consensus.cc:2781] T 9c40a6852649404f8928c2642e8ac3be P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:56.125056 4271 raft_consensus.cc:686] T 9c40a6852649404f8928c2642e8ac3be P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEADER]: Becoming Leader. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Running, Role: LEADER
I20240502 21:26:56.125378 4271 consensus_queue.cc:237] T 9c40a6852649404f8928c2642e8ac3be P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:56.115248 2230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c40a6852649404f8928c2642e8ac3be" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:26:56.126711 2230 raft_consensus.cc:2454] T 9c40a6852649404f8928c2642e8ac3be P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 0.
I20240502 21:26:56.127063 4366 ts_tablet_manager.cc:1398] T 9c40a6852649404f8928c2642e8ac3be P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20240502 21:26:56.127313 4366 tablet_bootstrap.cc:492] T b1d59ebecac34cf7b33424cc73cff030 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:56.128862 4366 tablet_bootstrap.cc:654] T b1d59ebecac34cf7b33424cc73cff030 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:56.120384 2225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c40a6852649404f8928c2642e8ac3be" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:26:56.130376 2225 raft_consensus.cc:3037] T 9c40a6852649404f8928c2642e8ac3be P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:56.132752 2225 raft_consensus.cc:2454] T 9c40a6852649404f8928c2642e8ac3be P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 1.
I20240502 21:26:56.134074 1964 raft_consensus.cc:1261] T 1022c4c6ee8642ca95250933a940fe7a P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:56.134819 3765 consensus_queue.cc:1035] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:56.135100 2094 raft_consensus.cc:1261] T 1022c4c6ee8642ca95250933a940fe7a P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:56.137487 3765 consensus_queue.cc:1035] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:56.139488 31667 raft_consensus.cc:1261] T 1022c4c6ee8642ca95250933a940fe7a P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:56.140002 3765 consensus_queue.cc:1035] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:56.139982 31395 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:26:56.145783 3504 raft_consensus.cc:482] T 1bf575f91df54d75a9ac519924fb62e7 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:56.145941 3504 raft_consensus.cc:504] T 1bf575f91df54d75a9ac519924fb62e7 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:56.146556 3504 leader_election.cc:288] T 1bf575f91df54d75a9ac519924fb62e7 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:56.147230 2225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bf575f91df54d75a9ac519924fb62e7" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:26:56.147321 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bf575f91df54d75a9ac519924fb62e7" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:56.147420 2225 raft_consensus.cc:2454] T 1bf575f91df54d75a9ac519924fb62e7 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:56.147485 31666 raft_consensus.cc:2454] T 1bf575f91df54d75a9ac519924fb62e7 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:56.147804 31862 leader_election.cc:302] T 1bf575f91df54d75a9ac519924fb62e7 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:56.148190 3504 raft_consensus.cc:2781] T 1bf575f91df54d75a9ac519924fb62e7 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:56.148322 3504 raft_consensus.cc:482] T 1bf575f91df54d75a9ac519924fb62e7 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:56.148415 3504 raft_consensus.cc:3037] T 1bf575f91df54d75a9ac519924fb62e7 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:56.152477 4188 raft_consensus.cc:1050] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f: attempting to promote NON_VOTER d1406e92aec84ac585304e57bd37a40d to VOTER
I20240502 21:26:56.153368 4188 consensus_queue.cc:237] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } }
I20240502 21:26:56.154660 1813 tablet_service.cc:1467] Processing CreateTablet for tablet fe7e65dd33b74bd4aaad54a5cf6a5626 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=df5c87578fa74cf98ecf89abaeb0acfa]), partition=
I20240502 21:26:56.155042 1813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe7e65dd33b74bd4aaad54a5cf6a5626. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:56.159718 31775 tablet_service.cc:1467] Processing CreateTablet for tablet fe7e65dd33b74bd4aaad54a5cf6a5626 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=df5c87578fa74cf98ecf89abaeb0acfa]), partition=
I20240502 21:26:56.160099 31775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe7e65dd33b74bd4aaad54a5cf6a5626. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:56.166474 4388 ts_tablet_manager.cc:1888] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:56.166584 4388 log.cc:1196] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal/wals/43c02342d1ce4b53870e0495116ef173
I20240502 21:26:56.167011 4388 ts_tablet_manager.cc:1909] T 43c02342d1ce4b53870e0495116ef173 P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting consensus metadata
I20240502 21:26:56.169775 31796 raft_consensus.cc:1261] T de2216a74f8344779a881651b66a85e5 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEARNER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:56.170323 1834 raft_consensus.cc:1261] T de2216a74f8344779a881651b66a85e5 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:56.170888 3774 consensus_queue.cc:1035] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:56.171378 3774 consensus_queue.cc:1035] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:56.172616 4389 ts_tablet_manager.cc:1888] T 43c02342d1ce4b53870e0495116ef173 P 9dec645b473f40578e726f3e50e219bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:56.172793 4389 log.cc:1196] T 43c02342d1ce4b53870e0495116ef173 P 9dec645b473f40578e726f3e50e219bf: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal/wals/43c02342d1ce4b53870e0495116ef173
I20240502 21:26:56.173303 4389 ts_tablet_manager.cc:1909] T 43c02342d1ce4b53870e0495116ef173 P 9dec645b473f40578e726f3e50e219bf: Deleting consensus metadata
I20240502 21:26:56.174062 31795 raft_consensus.cc:1261] T 0b8d119ee8474a40ac08df61df5b8423 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:56.175246 4366 tablet_bootstrap.cc:492] T b1d59ebecac34cf7b33424cc73cff030 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:56.175434 4366 ts_tablet_manager.cc:1367] T b1d59ebecac34cf7b33424cc73cff030 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.000s
I20240502 21:26:56.175624 31381 catalog_manager.cc:4830] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): tablet 43c02342d1ce4b53870e0495116ef173 (table rebalancer_dynamic_table_new_name [id=03638b9bda5347dcbf84b97a9589a9c6]) successfully deleted
I20240502 21:26:56.176076 4366 raft_consensus.cc:348] T b1d59ebecac34cf7b33424cc73cff030 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:56.176257 4366 raft_consensus.cc:374] T b1d59ebecac34cf7b33424cc73cff030 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:56.176342 4366 raft_consensus.cc:729] T b1d59ebecac34cf7b33424cc73cff030 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:56.176534 4366 consensus_queue.cc:260] T b1d59ebecac34cf7b33424cc73cff030 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:56.178361 4366 ts_tablet_manager.cc:1398] T b1d59ebecac34cf7b33424cc73cff030 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:26:56.179634 2095 raft_consensus.cc:1261] T de2216a74f8344779a881651b66a85e5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:56.180131 4324 consensus_queue.cc:1035] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:56.182374 31379 catalog_manager.cc:4830] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): tablet 43c02342d1ce4b53870e0495116ef173 (table rebalancer_dynamic_table_new_name [id=03638b9bda5347dcbf84b97a9589a9c6]) successfully deleted
I20240502 21:26:56.182745 31923 raft_consensus.cc:1261] T 0b8d119ee8474a40ac08df61df5b8423 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:56.183955 3765 raft_consensus.cc:2932] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:26:56.184691 3774 consensus_queue.cc:1035] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:56.185289 3774 consensus_queue.cc:1035] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:56.186194 31387 catalog_manager.cc:5484] T 9c40a6852649404f8928c2642e8ac3be P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135). New cstate: current_term: 1 leader_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:56.195237 31667 raft_consensus.cc:2932] T 1022c4c6ee8642ca95250933a940fe7a P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:26:56.197300 2097 raft_consensus.cc:2932] T 1022c4c6ee8642ca95250933a940fe7a P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:26:56.202905 2076 tablet_service.cc:1467] Processing CreateTablet for tablet fe7e65dd33b74bd4aaad54a5cf6a5626 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=df5c87578fa74cf98ecf89abaeb0acfa]), partition=
I20240502 21:26:56.203354 2076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe7e65dd33b74bd4aaad54a5cf6a5626. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:56.203503 31387 catalog_manager.cc:5484] T 1022c4c6ee8642ca95250933a940fe7a P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:26:56.215109 3859 raft_consensus.cc:1050] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada: attempting to promote NON_VOTER 9dec645b473f40578e726f3e50e219bf to VOTER
I20240502 21:26:56.215929 3859 consensus_queue.cc:237] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:26:56.216655 3504 raft_consensus.cc:504] T 1bf575f91df54d75a9ac519924fb62e7 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:56.218845 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bf575f91df54d75a9ac519924fb62e7" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:56.219053 31666 raft_consensus.cc:3037] T 1bf575f91df54d75a9ac519924fb62e7 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:56.222071 31666 raft_consensus.cc:2454] T 1bf575f91df54d75a9ac519924fb62e7 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:26:56.222599 31863 leader_election.cc:302] T 1bf575f91df54d75a9ac519924fb62e7 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:56.222960 3824 raft_consensus.cc:2781] T 1bf575f91df54d75a9ac519924fb62e7 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:56.223141 3824 raft_consensus.cc:686] T 1bf575f91df54d75a9ac519924fb62e7 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Running, Role: LEADER
I20240502 21:26:56.223459 3824 consensus_queue.cc:237] T 1bf575f91df54d75a9ac519924fb62e7 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:56.231786 3529 raft_consensus.cc:1050] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db: attempting to promote NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada to VOTER
I20240502 21:26:56.232911 3529 consensus_queue.cc:237] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } }
I20240502 21:26:56.240504 31923 raft_consensus.cc:1261] T b20df8896a2b423cb53c2c1a2a1b98e2 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:56.241020 4411 tablet_bootstrap.cc:492] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:56.241053 2225 raft_consensus.cc:1261] T b20df8896a2b423cb53c2c1a2a1b98e2 P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:56.241431 31795 raft_consensus.cc:1261] T b20df8896a2b423cb53c2c1a2a1b98e2 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:56.241580 3911 consensus_queue.cc:1035] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:56.241849 3774 raft_consensus.cc:2932] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:56.242539 4411 tablet_bootstrap.cc:654] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:56.243412 3911 consensus_queue.cc:1035] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 21:26:56.248184 2163 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.226.135:38569, user_credentials={real_user=slave}} blocked reactor thread for 59566.8us
I20240502 21:26:56.251892 1965 raft_consensus.cc:2932] T 1022c4c6ee8642ca95250933a940fe7a P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:26:56.253566 4188 consensus_queue.cc:1035] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:56.259183 31666 raft_consensus.cc:1261] T 577e7a586ad9459f8e77ce4b8f563c09 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:56.259671 2230 raft_consensus.cc:1261] T 577e7a586ad9459f8e77ce4b8f563c09 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:56.259706 4336 consensus_queue.cc:1035] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:56.262560 4337 consensus_queue.cc:1035] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:56.263636 3504 leader_election.cc:288] T 1bf575f91df54d75a9ac519924fb62e7 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Requested vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:56.265265 2095 raft_consensus.cc:1261] T 577e7a586ad9459f8e77ce4b8f563c09 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:56.265798 4336 consensus_queue.cc:1035] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:56.267619 4411 tablet_bootstrap.cc:492] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:56.268522 4411 ts_tablet_manager.cc:1367] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20240502 21:26:56.269683 4411 raft_consensus.cc:348] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:56.275297 4411 raft_consensus.cc:374] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:56.275694 31536 raft_consensus.cc:1203] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:26:56.276527 4411 raft_consensus.cc:729] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:56.278151 4411 consensus_queue.cc:260] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:56.283146 4411 ts_tablet_manager.cc:1398] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240502 21:26:56.289937 4423 tablet_bootstrap.cc:492] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:56.297003 2230 consensus_queue.cc:237] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } }
I20240502 21:26:56.299079 4423 tablet_bootstrap.cc:654] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:56.299346 1965 raft_consensus.cc:1261] T 1022c4c6ee8642ca95250933a940fe7a P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:56.299724 31667 raft_consensus.cc:1261] T 1022c4c6ee8642ca95250933a940fe7a P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:56.299921 4277 consensus_queue.cc:1035] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:56.300454 4407 consensus_queue.cc:1035] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:56.300163 4336 raft_consensus.cc:2932] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:26:56.301134 31667 raft_consensus.cc:2932] T 577e7a586ad9459f8e77ce4b8f563c09 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:26:56.304723 2224 raft_consensus.cc:2932] T 577e7a586ad9459f8e77ce4b8f563c09 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:26:56.315693 3765 raft_consensus.cc:2932] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:26:56.316915 1964 raft_consensus.cc:2932] T 1022c4c6ee8642ca95250933a940fe7a P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:26:56.318917 31381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1022c4c6ee8642ca95250933a940fe7a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:56.323652 4123 raft_consensus.cc:1050] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b: attempting to promote NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada to VOTER
I20240502 21:26:56.322827 31666 raft_consensus.cc:2932] T 1022c4c6ee8642ca95250933a940fe7a P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:26:56.324847 4123 consensus_queue.cc:237] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } }
I20240502 21:26:56.331293 31386 catalog_manager.cc:5484] T 1022c4c6ee8642ca95250933a940fe7a P 9dec645b473f40578e726f3e50e219bf reported cstate change: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:26:56.332840 2230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bf575f91df54d75a9ac519924fb62e7" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:26:56.333096 2230 raft_consensus.cc:3037] T 1bf575f91df54d75a9ac519924fb62e7 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:56.333576 31388 catalog_manager.cc:5484] T 577e7a586ad9459f8e77ce4b8f563c09 P d1406e92aec84ac585304e57bd37a40d reported cstate change: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:26:56.335788 2230 raft_consensus.cc:2454] T 1bf575f91df54d75a9ac519924fb62e7 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:26:56.339345 2077 tablet_service.cc:1514] Processing DeleteTablet for tablet 1022c4c6ee8642ca95250933a940fe7a with delete_type TABLET_DATA_TOMBSTONED (TS 1ade0354c44c4c37ae480c7a94b11e1b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36740
I20240502 21:26:56.339634 4388 tablet_replica.cc:330] T 1022c4c6ee8642ca95250933a940fe7a P 1ade0354c44c4c37ae480c7a94b11e1b: stopping tablet replica
I20240502 21:26:56.339833 4388 raft_consensus.cc:2229] T 1022c4c6ee8642ca95250933a940fe7a P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:56.339998 4388 raft_consensus.cc:2258] T 1022c4c6ee8642ca95250933a940fe7a P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:56.340878 4388 ts_tablet_manager.cc:1875] T 1022c4c6ee8642ca95250933a940fe7a P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:56.344573 4337 raft_consensus.cc:482] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:56.344754 4337 raft_consensus.cc:504] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:56.345489 4337 leader_election.cc:288] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:56.346292 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe7e65dd33b74bd4aaad54a5cf6a5626" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
W20240502 21:26:56.346570 1769 leader_election.cc:341] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): Not found: Tablet not found: fe7e65dd33b74bd4aaad54a5cf6a5626
W20240502 21:26:56.347256 1769 leader_election.cc:341] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:56.347433 1769 leader_election.cc:302] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db; no voters: 08d7a0680d244b55b93d17f9ab57b84c, 1ade0354c44c4c37ae480c7a94b11e1b
I20240502 21:26:56.347924 4337 raft_consensus.cc:2726] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:26:56.349380 4365 tablet_bootstrap.cc:492] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:56.350651 4365 tablet_bootstrap.cc:654] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:56.353879 2095 raft_consensus.cc:2932] T 577e7a586ad9459f8e77ce4b8f563c09 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:26:56.356683 1833 consensus_queue.cc:237] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } }
I20240502 21:26:56.358843 31667 raft_consensus.cc:1261] T 577e7a586ad9459f8e77ce4b8f563c09 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:56.358973 31665 raft_consensus.cc:1261] T 46279270b2334633906af57c4b43e969 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Refusing update from remote peer 1ade0354c44c4c37ae480c7a94b11e1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:56.359470 4336 consensus_queue.cc:1035] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:56.359962 1964 raft_consensus.cc:1261] T 46279270b2334633906af57c4b43e969 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer 1ade0354c44c4c37ae480c7a94b11e1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:56.360484 4271 consensus_queue.cc:1035] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:56.360721 2224 raft_consensus.cc:1261] T 577e7a586ad9459f8e77ce4b8f563c09 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:56.360939 4388 ts_tablet_manager.cc:1888] T 1022c4c6ee8642ca95250933a940fe7a P 1ade0354c44c4c37ae480c7a94b11e1b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:26:56.361080 4388 log.cc:1196] T 1022c4c6ee8642ca95250933a940fe7a P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal/wals/1022c4c6ee8642ca95250933a940fe7a
I20240502 21:26:56.361016 4271 consensus_queue.cc:1035] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:56.361696 4336 consensus_queue.cc:1035] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:56.361874 31381 catalog_manager.cc:4830] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): tablet 1022c4c6ee8642ca95250933a940fe7a (table rebalancer_extra_table_4 [id=5fea5cf0a15d489c9c9a523859a65476]) successfully deleted
I20240502 21:26:56.365784 4336 raft_consensus.cc:2932] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:26:56.370075 31381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 577e7a586ad9459f8e77ce4b8f563c09 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:56.370682 2224 raft_consensus.cc:2932] T 577e7a586ad9459f8e77ce4b8f563c09 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:26:56.379817 31665 raft_consensus.cc:2932] T 577e7a586ad9459f8e77ce4b8f563c09 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
W20240502 21:26:56.385942 31602 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.226.136:34907, user_credentials={real_user=slave}} blocked reactor thread for 136500us
I20240502 21:26:56.386313 2230 raft_consensus.cc:1261] T 46279270b2334633906af57c4b43e969 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 1ade0354c44c4c37ae480c7a94b11e1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:56.386139 1965 raft_consensus.cc:2932] T 0b8d119ee8474a40ac08df61df5b8423 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:56.386957 4123 consensus_queue.cc:1035] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:56.388337 31795 raft_consensus.cc:2932] T 0b8d119ee8474a40ac08df61df5b8423 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:56.392258 31924 raft_consensus.cc:2932] T 0b8d119ee8474a40ac08df61df5b8423 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:56.394547 4188 raft_consensus.cc:1050] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f: attempting to promote NON_VOTER d1406e92aec84ac585304e57bd37a40d to VOTER
I20240502 21:26:56.395494 4188 consensus_queue.cc:237] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } }
I20240502 21:26:56.403729 4365 tablet_bootstrap.cc:492] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:56.403915 4365 ts_tablet_manager.cc:1367] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.055s user 0.004s sys 0.008s
I20240502 21:26:56.403770 4271 raft_consensus.cc:2932] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:26:56.406069 4423 tablet_bootstrap.cc:492] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:56.405794 1834 raft_consensus.cc:2932] T de2216a74f8344779a881651b66a85e5 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:26:56.406204 4423 ts_tablet_manager.cc:1367] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.116s user 0.000s sys 0.012s
I20240502 21:26:56.407195 2230 raft_consensus.cc:2932] T 46279270b2334633906af57c4b43e969 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:26:56.408710 1965 raft_consensus.cc:2932] T 46279270b2334633906af57c4b43e969 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:26:56.411356 31796 raft_consensus.cc:2932] T de2216a74f8344779a881651b66a85e5 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:26:56.422412 4365 raft_consensus.cc:348] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:56.422595 4365 raft_consensus.cc:374] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:56.422683 4365 raft_consensus.cc:729] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:56.416679 4324 raft_consensus.cc:2932] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:26:56.427795 3911 raft_consensus.cc:2932] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:56.431275 2097 raft_consensus.cc:2932] T de2216a74f8344779a881651b66a85e5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:26:56.438315 31923 raft_consensus.cc:1261] T 1ed225bdefc64ddd8eef2109bfd969df P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:56.438372 2224 raft_consensus.cc:2932] T b20df8896a2b423cb53c2c1a2a1b98e2 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:56.439013 4433 consensus_queue.cc:1035] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:56.441733 4423 raft_consensus.cc:348] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:56.441920 4423 raft_consensus.cc:374] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:56.442015 4423 raft_consensus.cc:729] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:56.442240 4423 consensus_queue.cc:260] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:56.442925 4423 ts_tablet_manager.cc:1398] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.037s user 0.002s sys 0.000s
I20240502 21:26:56.444205 31797 raft_consensus.cc:2932] T b20df8896a2b423cb53c2c1a2a1b98e2 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:56.445223 31667 raft_consensus.cc:2932] T 46279270b2334633906af57c4b43e969 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:26:56.449885 31924 raft_consensus.cc:2932] T b20df8896a2b423cb53c2c1a2a1b98e2 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:56.451498 31391 catalog_manager.cc:5484] T 0b8d119ee8474a40ac08df61df5b8423 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index 4 to 5, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:56.451843 31392 catalog_manager.cc:5484] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:56.454138 4365 consensus_queue.cc:260] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:56.454741 4365 ts_tablet_manager.cc:1398] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.051s user 0.000s sys 0.001s
I20240502 21:26:56.460345 31797 raft_consensus.cc:1261] T 1ed225bdefc64ddd8eef2109bfd969df P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:56.461447 2230 raft_consensus.cc:1261] T 1ed225bdefc64ddd8eef2109bfd969df P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:56.461158 31393 catalog_manager.cc:5484] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: config changed from index 4 to 5, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:56.462268 4188 consensus_queue.cc:1035] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:56.462568 3911 consensus_queue.cc:1035] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:56.467525 31394 catalog_manager.cc:5484] T 1bf575f91df54d75a9ac519924fb62e7 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132). New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:56.452245 31391 catalog_manager.cc:5484] T de2216a74f8344779a881651b66a85e5 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index 2 to 3, 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:26:56.472954 4439 raft_consensus.cc:482] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:56.473212 4439 raft_consensus.cc:504] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:56.474735 4439 leader_election.cc:288] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:56.474748 1834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe7e65dd33b74bd4aaad54a5cf6a5626" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:26:56.474968 31796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe7e65dd33b74bd4aaad54a5cf6a5626" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:56.475219 31796 raft_consensus.cc:2454] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 0.
I20240502 21:26:56.475395 1834 raft_consensus.cc:2454] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 0.
I20240502 21:26:56.475672 2032 leader_election.cc:302] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 1ade0354c44c4c37ae480c7a94b11e1b; no voters:
I20240502 21:26:56.476166 4439 raft_consensus.cc:2781] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:56.476279 4439 raft_consensus.cc:482] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:56.476377 4439 raft_consensus.cc:3037] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:56.479040 4271 consensus_queue.cc:1035] T b1d59ebecac34cf7b33424cc73cff030 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:56.485549 31533 raft_consensus.cc:1261] T 182b3f15c06c4c07ade0a8854d3ecafd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:26:56.485615 2095 consensus_queue.cc:237] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } }
I20240502 21:26:56.486567 4417 consensus_queue.cc:1035] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:26:56.488516 4439 raft_consensus.cc:504] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:56.488543 31536 consensus_queue.cc:237] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } }
I20240502 21:26:56.489485 1834 raft_consensus.cc:1261] T 182b3f15c06c4c07ade0a8854d3ecafd P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:26:56.490398 4417 consensus_queue.cc:1035] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:26:56.491518 1832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe7e65dd33b74bd4aaad54a5cf6a5626" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:26:56.491760 1832 raft_consensus.cc:3037] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:56.491991 4409 raft_consensus.cc:482] T b1d59ebecac34cf7b33424cc73cff030 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:56.492126 4409 raft_consensus.cc:504] T b1d59ebecac34cf7b33424cc73cff030 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:56.492789 4409 leader_election.cc:288] T b1d59ebecac34cf7b33424cc73cff030 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:26:56.492841 1965 raft_consensus.cc:1261] T 182b3f15c06c4c07ade0a8854d3ecafd P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:26:56.494485 1832 raft_consensus.cc:2454] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 1.
I20240502 21:26:56.494567 31793 raft_consensus.cc:1261] T b20df8896a2b423cb53c2c1a2a1b98e2 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:56.494740 31386 catalog_manager.cc:5484] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db reported cstate change: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:56.495466 4417 consensus_queue.cc:1035] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:26:56.497294 31647 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:26:56.497565 1966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1d59ebecac34cf7b33424cc73cff030" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:26:56.498143 1962 raft_consensus.cc:1261] T 46279270b2334633906af57c4b43e969 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer 1ade0354c44c4c37ae480c7a94b11e1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:56.498214 4188 consensus_queue.cc:1035] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:56.500248 2032 leader_election.cc:302] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 1e04157b98dd4fad98858c290b73c3db; no voters:
I20240502 21:26:56.500531 4268 consensus_queue.cc:1035] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:56.501080 4268 raft_consensus.cc:2781] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:56.501219 4268 raft_consensus.cc:686] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEADER]: Becoming Leader. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Running, Role: LEADER
I20240502 21:26:56.501488 4268 consensus_queue.cc:237] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:56.504814 2095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1d59ebecac34cf7b33424cc73cff030" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:56.505486 2163 leader_election.cc:302] T b1d59ebecac34cf7b33424cc73cff030 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1406e92aec84ac585304e57bd37a40d; no voters: 1ade0354c44c4c37ae480c7a94b11e1b, 9dec645b473f40578e726f3e50e219bf
I20240502 21:26:56.505743 4409 raft_consensus.cc:2726] T b1d59ebecac34cf7b33424cc73cff030 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:26:56.506891 4123 raft_consensus.cc:2932] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:26:56.508221 2224 raft_consensus.cc:1261] T b20df8896a2b423cb53c2c1a2a1b98e2 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:56.509683 4433 consensus_queue.cc:1035] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:56.510527 1962 raft_consensus.cc:2932] T 46279270b2334633906af57c4b43e969 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:26:56.510830 4188 raft_consensus.cc:2932] T b20df8896a2b423cb53c2c1a2a1b98e2 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:56.515020 2076 tablet_service.cc:1514] Processing DeleteTablet for tablet 577e7a586ad9459f8e77ce4b8f563c09 with delete_type TABLET_DATA_TOMBSTONED (TS 1ade0354c44c4c37ae480c7a94b11e1b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36740
I20240502 21:26:56.515647 31382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b20df8896a2b423cb53c2c1a2a1b98e2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:56.516034 4388 tablet_replica.cc:330] T 577e7a586ad9459f8e77ce4b8f563c09 P 1ade0354c44c4c37ae480c7a94b11e1b: stopping tablet replica
I20240502 21:26:56.516253 4388 raft_consensus.cc:2229] T 577e7a586ad9459f8e77ce4b8f563c09 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:56.515872 31796 raft_consensus.cc:2932] T b20df8896a2b423cb53c2c1a2a1b98e2 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:56.516431 4388 raft_consensus.cc:2258] T 577e7a586ad9459f8e77ce4b8f563c09 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:56.517393 31381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 46279270b2334633906af57c4b43e969 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:56.517429 4388 ts_tablet_manager.cc:1875] T 577e7a586ad9459f8e77ce4b8f563c09 P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:56.521916 2225 raft_consensus.cc:2932] T b20df8896a2b423cb53c2c1a2a1b98e2 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:56.524622 31664 consensus_queue.cc:237] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } }
I20240502 21:26:56.524725 31665 consensus_queue.cc:237] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:26:56.526384 31667 raft_consensus.cc:1261] T 46279270b2334633906af57c4b43e969 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 1ade0354c44c4c37ae480c7a94b11e1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:56.527456 2095 raft_consensus.cc:1261] T de2216a74f8344779a881651b66a85e5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:56.527577 4271 consensus_queue.cc:1035] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:56.527644 31924 raft_consensus.cc:1261] T 0b8d119ee8474a40ac08df61df5b8423 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:56.528146 4417 consensus_queue.cc:1035] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:56.528235 31796 raft_consensus.cc:1261] T de2216a74f8344779a881651b66a85e5 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:56.528663 4417 consensus_queue.cc:1035] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:56.529034 31665 raft_consensus.cc:2932] T 46279270b2334633906af57c4b43e969 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:26:56.530161 31389 catalog_manager.cc:5484] T b20df8896a2b423cb53c2c1a2a1b98e2 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index 5 to 6, VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) evicted. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:56.531280 4439 leader_election.cc:288] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 election: Requested vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:56.531626 31796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe7e65dd33b74bd4aaad54a5cf6a5626" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:56.531695 3774 consensus_queue.cc:1035] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:56.531857 31796 raft_consensus.cc:3037] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:56.532205 1962 raft_consensus.cc:1261] T 0b8d119ee8474a40ac08df61df5b8423 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:56.533371 2095 raft_consensus.cc:2932] T de2216a74f8344779a881651b66a85e5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:26:56.534546 31796 raft_consensus.cc:2454] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 1.
I20240502 21:26:56.535687 4388 ts_tablet_manager.cc:1888] T 577e7a586ad9459f8e77ce4b8f563c09 P 1ade0354c44c4c37ae480c7a94b11e1b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:26:56.535847 4388 log.cc:1196] T 577e7a586ad9459f8e77ce4b8f563c09 P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal/wals/577e7a586ad9459f8e77ce4b8f563c09
I20240502 21:26:56.536681 31381 catalog_manager.cc:4830] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): tablet 577e7a586ad9459f8e77ce4b8f563c09 (table rebalancer_extra_table_6 [id=86898baa76944823b3c5e8aba98d8772]) successfully deleted
I20240502 21:26:56.539543 31905 tablet_service.cc:1514] Processing DeleteTablet for tablet b20df8896a2b423cb53c2c1a2a1b98e2 with delete_type TABLET_DATA_TOMBSTONED (TS 2d50f33557944bb480ea32199d9ee4f1 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47824
I20240502 21:26:56.541597 3774 raft_consensus.cc:2932] T de2216a74f8344779a881651b66a85e5 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:26:56.542546 31391 catalog_manager.cc:5484] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: config changed from index 3 to 4, VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) evicted. New cstate: current_term: 1 leader_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:56.545473 4324 consensus_queue.cc:1035] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20240502 21:26:56.545857 31796 raft_consensus.cc:2932] T de2216a74f8344779a881651b66a85e5 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:26:56.547727 31924 raft_consensus.cc:2932] T 0b8d119ee8474a40ac08df61df5b8423 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:56.548601 1962 raft_consensus.cc:2932] T 0b8d119ee8474a40ac08df61df5b8423 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:56.546434 4324 raft_consensus.cc:2932] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:56.555404 31389 catalog_manager.cc:5484] T 0b8d119ee8474a40ac08df61df5b8423 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: config changed from index 5 to 6, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:56.558799 4433 raft_consensus.cc:2932] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:56.555703 31382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet de2216a74f8344779a881651b66a85e5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:56.560314 31382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0b8d119ee8474a40ac08df61df5b8423 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:56.562011 31391 catalog_manager.cc:5484] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135). New cstate: current_term: 1 leader_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:56.566375 31924 raft_consensus.cc:2932] T 1ed225bdefc64ddd8eef2109bfd969df P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:56.572623 31392 catalog_manager.cc:5484] T 1ed225bdefc64ddd8eef2109bfd969df P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: config changed from index 4 to 5, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:56.572718 31795 raft_consensus.cc:2932] T 1ed225bdefc64ddd8eef2109bfd969df P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:56.575842 2224 raft_consensus.cc:3037] T b1d59ebecac34cf7b33424cc73cff030 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:56.578507 3504 raft_consensus.cc:1050] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1: attempting to promote NON_VOTER 1e04157b98dd4fad98858c290b73c3db to VOTER
I20240502 21:26:56.579275 3504 consensus_queue.cc:237] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } }
I20240502 21:26:56.580732 4268 consensus_queue.cc:1035] T b1d59ebecac34cf7b33424cc73cff030 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:56.582438 2208 tablet_service.cc:1514] Processing DeleteTablet for tablet 46279270b2334633906af57c4b43e969 with delete_type TABLET_DATA_TOMBSTONED (TS d1406e92aec84ac585304e57bd37a40d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36984
I20240502 21:26:56.582803 4387 tablet_replica.cc:330] T 46279270b2334633906af57c4b43e969 P d1406e92aec84ac585304e57bd37a40d: stopping tablet replica
I20240502 21:26:56.583056 4387 raft_consensus.cc:2229] T 46279270b2334633906af57c4b43e969 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:56.583281 4387 raft_consensus.cc:2258] T 46279270b2334633906af57c4b43e969 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:56.583902 31775 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b8d119ee8474a40ac08df61df5b8423 with delete_type TABLET_DATA_TOMBSTONED (TS 08d7a0680d244b55b93d17f9ab57b84c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51596
I20240502 21:26:56.585958 4387 ts_tablet_manager.cc:1875] T 46279270b2334633906af57c4b43e969 P d1406e92aec84ac585304e57bd37a40d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:56.589988 31533 consensus_queue.cc:237] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } }
I20240502 21:26:56.591202 31391 catalog_manager.cc:5484] T de2216a74f8344779a881651b66a85e5 P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: config changed from index 3 to 4, VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) evicted. New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:26:56.594691 4432 raft_consensus.cc:2932] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:56.595448 1832 raft_consensus.cc:2932] T 182b3f15c06c4c07ade0a8854d3ecafd P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:56.598486 4454 tablet_replica.cc:330] T b20df8896a2b423cb53c2c1a2a1b98e2 P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:26:56.598747 4454 raft_consensus.cc:2229] T b20df8896a2b423cb53c2c1a2a1b98e2 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:56.599050 4454 raft_consensus.cc:2258] T b20df8896a2b423cb53c2c1a2a1b98e2 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:56.594794 1965 raft_consensus.cc:2932] T 182b3f15c06c4c07ade0a8854d3ecafd P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:56.601842 31386 catalog_manager.cc:5484] T 182b3f15c06c4c07ade0a8854d3ecafd P 1e04157b98dd4fad98858c290b73c3db reported cstate change: config changed from index 7 to 8, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:56.610689 1813 tablet_service.cc:1514] Processing DeleteTablet for tablet de2216a74f8344779a881651b66a85e5 with delete_type TABLET_DATA_TOMBSTONED (TS 1e04157b98dd4fad98858c290b73c3db not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47510
I20240502 21:26:56.610729 31797 raft_consensus.cc:1261] T 1ed225bdefc64ddd8eef2109bfd969df P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:56.611650 4430 consensus_queue.cc:1035] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:56.614373 4430 raft_consensus.cc:2932] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:56.620342 31797 raft_consensus.cc:2932] T 1ed225bdefc64ddd8eef2109bfd969df P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:56.626776 4456 tablet_replica.cc:330] T 0b8d119ee8474a40ac08df61df5b8423 P 08d7a0680d244b55b93d17f9ab57b84c: stopping tablet replica
I20240502 21:26:56.629616 31382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1ed225bdefc64ddd8eef2109bfd969df with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:56.632644 31387 catalog_manager.cc:5484] T 1ed225bdefc64ddd8eef2109bfd969df P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index 5 to 6, VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) evicted. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:56.633431 4456 raft_consensus.cc:2229] T 0b8d119ee8474a40ac08df61df5b8423 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:56.634600 4456 raft_consensus.cc:2258] T 0b8d119ee8474a40ac08df61df5b8423 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:56.637246 31905 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ed225bdefc64ddd8eef2109bfd969df with delete_type TABLET_DATA_TOMBSTONED (TS 2d50f33557944bb480ea32199d9ee4f1 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47824
I20240502 21:26:56.640565 4456 ts_tablet_manager.cc:1875] T 0b8d119ee8474a40ac08df61df5b8423 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:56.640554 31533 raft_consensus.cc:2932] T 182b3f15c06c4c07ade0a8854d3ecafd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:56.647764 31667 consensus_queue.cc:237] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:26:56.650431 1832 raft_consensus.cc:1261] T 182b3f15c06c4c07ade0a8854d3ecafd P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:26:56.651064 4324 consensus_queue.cc:1035] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:26:56.652801 1965 raft_consensus.cc:1261] T 182b3f15c06c4c07ade0a8854d3ecafd P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:26:56.653700 4428 consensus_queue.cc:1035] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:26:56.654201 4428 raft_consensus.cc:2932] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:56.658331 1962 raft_consensus.cc:2932] T 182b3f15c06c4c07ade0a8854d3ecafd P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:56.662170 1832 raft_consensus.cc:2932] T 182b3f15c06c4c07ade0a8854d3ecafd P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:56.665199 4456 ts_tablet_manager.cc:1888] T 0b8d119ee8474a40ac08df61df5b8423 P 08d7a0680d244b55b93d17f9ab57b84c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:26:56.670614 31382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 182b3f15c06c4c07ade0a8854d3ecafd with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:56.673501 4461 tablet_replica.cc:330] T de2216a74f8344779a881651b66a85e5 P 1e04157b98dd4fad98858c290b73c3db: stopping tablet replica
I20240502 21:26:56.673686 4461 raft_consensus.cc:2229] T de2216a74f8344779a881651b66a85e5 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:56.673861 4461 raft_consensus.cc:2258] T de2216a74f8344779a881651b66a85e5 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:56.674127 31391 catalog_manager.cc:5484] T 182b3f15c06c4c07ade0a8854d3ecafd P 1e04157b98dd4fad98858c290b73c3db reported cstate change: config changed from index 8 to 9, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:26:56.675053 4456 log.cc:1196] T 0b8d119ee8474a40ac08df61df5b8423 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/wals/0b8d119ee8474a40ac08df61df5b8423
I20240502 21:26:56.675954 1832 raft_consensus.cc:1261] T 83904952a5a541ecac508449e23e70dd P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:26:56.676515 3824 consensus_queue.cc:1035] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:26:56.680342 31516 tablet_service.cc:1514] Processing DeleteTablet for tablet 182b3f15c06c4c07ade0a8854d3ecafd with delete_type TABLET_DATA_TOMBSTONED (TS a44f41b0b6644fb6aa29476aed8c964f not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:45194
I20240502 21:26:56.684877 4454 ts_tablet_manager.cc:1875] T b20df8896a2b423cb53c2c1a2a1b98e2 P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:56.685135 4461 ts_tablet_manager.cc:1875] T de2216a74f8344779a881651b66a85e5 P 1e04157b98dd4fad98858c290b73c3db: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:56.702533 2097 raft_consensus.cc:1261] T 83904952a5a541ecac508449e23e70dd P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:26:56.703572 3824 consensus_queue.cc:1035] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:26:56.705094 31795 raft_consensus.cc:1261] T 83904952a5a541ecac508449e23e70dd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:26:56.695070 2230 raft_consensus.cc:2932] T 1ed225bdefc64ddd8eef2109bfd969df P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:56.707412 31381 catalog_manager.cc:4830] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): tablet 0b8d119ee8474a40ac08df61df5b8423 (table rebalance_test_table_15 [id=7ff12e5a4c674a5fb7e07cb593390fd1]) successfully deleted
I20240502 21:26:56.711905 4454 ts_tablet_manager.cc:1888] T b20df8896a2b423cb53c2c1a2a1b98e2 P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:26:56.712013 4387 ts_tablet_manager.cc:1888] T 46279270b2334633906af57c4b43e969 P d1406e92aec84ac585304e57bd37a40d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:26:56.713631 4461 ts_tablet_manager.cc:1888] T de2216a74f8344779a881651b66a85e5 P 1e04157b98dd4fad98858c290b73c3db: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:26:56.713938 4454 log.cc:1196] T b20df8896a2b423cb53c2c1a2a1b98e2 P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/b20df8896a2b423cb53c2c1a2a1b98e2
I20240502 21:26:56.714802 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet b20df8896a2b423cb53c2c1a2a1b98e2 (table rebalance_test_table_3 [id=59305f5894b34d0085a6cada5a676ca4]) successfully deleted
I20240502 21:26:56.710196 3504 consensus_queue.cc:1035] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.006s
I20240502 21:26:56.716006 31391 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:26:56.720268 4387 log.cc:1196] T 46279270b2334633906af57c4b43e969 P d1406e92aec84ac585304e57bd37a40d: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal/wals/46279270b2334633906af57c4b43e969
I20240502 21:26:56.721094 31381 catalog_manager.cc:4830] TS d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): tablet 46279270b2334633906af57c4b43e969 (table rebalancer_extra_table_8 [id=4cfb4e8589a24f279c0e7f552012212c]) successfully deleted
I20240502 21:26:56.722904 2226 raft_consensus.cc:1261] T 1ed225bdefc64ddd8eef2109bfd969df P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:56.725216 4430 consensus_queue.cc:1035] T 1ed225bdefc64ddd8eef2109bfd969df P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:56.727357 2230 raft_consensus.cc:2932] T 1ed225bdefc64ddd8eef2109bfd969df P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:26:56.733680 31796 raft_consensus.cc:1261] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 1ade0354c44c4c37ae480c7a94b11e1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:56.734611 4439 consensus_queue.cc:1035] T 9c40a6852649404f8928c2642e8ac3be P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:56.735646 3504 consensus_queue.cc:1035] T 1bf575f91df54d75a9ac519924fb62e7 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:56.741367 3824 consensus_queue.cc:1035] T 1bf575f91df54d75a9ac519924fb62e7 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:56.739459 1832 raft_consensus.cc:1261] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer 1ade0354c44c4c37ae480c7a94b11e1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:56.748767 4461 log.cc:1196] T de2216a74f8344779a881651b66a85e5 P 1e04157b98dd4fad98858c290b73c3db: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal/wals/de2216a74f8344779a881651b66a85e5
I20240502 21:26:56.749675 31381 catalog_manager.cc:4830] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): tablet de2216a74f8344779a881651b66a85e5 (table rebalancer_extra_table_12 [id=daf389fa76d84011ad57fe7156cfa6a8]) successfully deleted
I20240502 21:26:56.750315 4454 tablet_replica.cc:330] T 1ed225bdefc64ddd8eef2109bfd969df P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:26:56.750542 4454 raft_consensus.cc:2229] T 1ed225bdefc64ddd8eef2109bfd969df P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:56.750774 4454 raft_consensus.cc:2258] T 1ed225bdefc64ddd8eef2109bfd969df P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:56.753463 4454 ts_tablet_manager.cc:1875] T 1ed225bdefc64ddd8eef2109bfd969df P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:56.761646 4454 ts_tablet_manager.cc:1888] T 1ed225bdefc64ddd8eef2109bfd969df P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:26:56.761945 4454 log.cc:1196] T 1ed225bdefc64ddd8eef2109bfd969df P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/1ed225bdefc64ddd8eef2109bfd969df
I20240502 21:26:56.762657 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet 1ed225bdefc64ddd8eef2109bfd969df (table rebalance_test_table_9 [id=5eee55eb24e94c988b4ddbfe2f8442d3]) successfully deleted
I20240502 21:26:56.766232 4268 consensus_queue.cc:1035] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:56.774760 4268 consensus_queue.cc:1035] T 9c40a6852649404f8928c2642e8ac3be P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:56.780665 4268 consensus_queue.cc:1035] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:56.786644 1813 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:26:56.789639 1946 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:26:56.813585 4485 raft_consensus.cc:2932] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:56.846212 31514 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:26:56.849283 31775 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:26:56.868207 1834 raft_consensus.cc:2932] T 83904952a5a541ecac508449e23e70dd P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:56.880319 4469 tablet_replica.cc:330] T 182b3f15c06c4c07ade0a8854d3ecafd P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:56.880647 4469 raft_consensus.cc:2229] T 182b3f15c06c4c07ade0a8854d3ecafd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:56.881071 4469 raft_consensus.cc:2258] T 182b3f15c06c4c07ade0a8854d3ecafd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:56.882191 4469 ts_tablet_manager.cc:1875] T 182b3f15c06c4c07ade0a8854d3ecafd P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:56.891355 31387 catalog_manager.cc:5484] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: config changed from index 7 to 8, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:56.896679 4407 raft_consensus.cc:1050] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d: attempting to promote NON_VOTER a44f41b0b6644fb6aa29476aed8c964f to VOTER
I20240502 21:26:56.897428 4407 consensus_queue.cc:237] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } }
I20240502 21:26:56.901738 2076 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:26:56.901829 31391 catalog_manager.cc:5576] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569) does not have the latest schema for tablet fe7e65dd33b74bd4aaad54a5cf6a5626 (table rebalancer_dynamic_table [id=df5c87578fa74cf98ecf89abaeb0acfa]). Expected version 1 got 0
I20240502 21:26:56.907589 2097 raft_consensus.cc:2932] T 83904952a5a541ecac508449e23e70dd P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:56.916883 31533 raft_consensus.cc:1261] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEARNER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:56.918596 4243 consensus_queue.cc:1035] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:56.923087 1965 raft_consensus.cc:1261] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:56.923628 4243 consensus_queue.cc:1035] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:56.925889 31387 catalog_manager.cc:5576] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863) does not have the latest schema for tablet fe7e65dd33b74bd4aaad54a5cf6a5626 (table rebalancer_dynamic_table [id=df5c87578fa74cf98ecf89abaeb0acfa]). Expected version 1 got 0
I20240502 21:26:56.943192 4243 raft_consensus.cc:2932] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:56.947791 1965 raft_consensus.cc:2932] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:56.948333 31536 raft_consensus.cc:2932] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:56.952575 1832 raft_consensus.cc:1261] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:56.956341 31387 catalog_manager.cc:5484] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: config changed from index 2 to 3, a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:56.960312 4409 consensus_queue.cc:1035] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:56.963820 31923 consensus_queue.cc:237] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } }
I20240502 21:26:56.966955 1833 raft_consensus.cc:1261] T 83904952a5a541ecac508449e23e70dd P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:26:56.968627 4489 consensus_queue.cc:1035] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:26:56.970147 31389 catalog_manager.cc:5576] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807) does not have the latest schema for tablet fe7e65dd33b74bd4aaad54a5cf6a5626 (table rebalancer_dynamic_table [id=df5c87578fa74cf98ecf89abaeb0acfa]). Expected version 1 got 0
I20240502 21:26:56.974371 4488 raft_consensus.cc:2932] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:56.977042 1833 raft_consensus.cc:2932] T 83904952a5a541ecac508449e23e70dd P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:56.980803 2208 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:26:56.987108 31382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 83904952a5a541ecac508449e23e70dd with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:56.990159 31392 catalog_manager.cc:5484] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: config changed from index 8 to 9, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:56.996718 31775 tablet_service.cc:1514] Processing DeleteTablet for tablet 83904952a5a541ecac508449e23e70dd with delete_type TABLET_DATA_TOMBSTONED (TS 08d7a0680d244b55b93d17f9ab57b84c not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51596
I20240502 21:26:56.997038 4456 tablet_replica.cc:330] T 83904952a5a541ecac508449e23e70dd P 08d7a0680d244b55b93d17f9ab57b84c: stopping tablet replica
I20240502 21:26:56.997231 4456 raft_consensus.cc:2229] T 83904952a5a541ecac508449e23e70dd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:56.997529 4456 pending_rounds.cc:62] T 83904952a5a541ecac508449e23e70dd P 08d7a0680d244b55b93d17f9ab57b84c: Trying to abort 1 pending ops.
I20240502 21:26:56.997581 4456 pending_rounds.cc:68] T 83904952a5a541ecac508449e23e70dd P 08d7a0680d244b55b93d17f9ab57b84c: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7023350647106809856 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "83904952a5a541ecac508449e23e70dd" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } } }
I20240502 21:26:56.997907 4456 raft_consensus.cc:2894] T 83904952a5a541ecac508449e23e70dd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240502 21:26:56.997992 4456 raft_consensus.cc:2866] T 83904952a5a541ecac508449e23e70dd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 21:26:56.998076 4456 raft_consensus.cc:2258] T 83904952a5a541ecac508449e23e70dd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:57.000010 4456 ts_tablet_manager.cc:1875] T 83904952a5a541ecac508449e23e70dd P 08d7a0680d244b55b93d17f9ab57b84c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:57.005025 31905 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:26:57.012936 4456 ts_tablet_manager.cc:1888] T 83904952a5a541ecac508449e23e70dd P 08d7a0680d244b55b93d17f9ab57b84c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:26:57.013064 4456 log.cc:1196] T 83904952a5a541ecac508449e23e70dd P 08d7a0680d244b55b93d17f9ab57b84c: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/wals/83904952a5a541ecac508449e23e70dd
I20240502 21:26:57.013793 31381 catalog_manager.cc:4830] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): tablet 83904952a5a541ecac508449e23e70dd (table rebalance_test_table_15 [id=7ff12e5a4c674a5fb7e07cb593390fd1]) successfully deleted
I20240502 21:26:57.017139 1832 raft_consensus.cc:2932] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:57.046418 4497 tablet.cc:1722] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:57.047160 4497 tablet.cc:1620] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1e04157b98dd4fad98858c290b73c3db: MemRowSet was empty: no flush needed.
I20240502 21:26:57.047938 4496 tablet.cc:1722] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:57.048630 4496 tablet.cc:1620] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b: MemRowSet was empty: no flush needed.
I20240502 21:26:57.048861 2094 raft_consensus.cc:1261] T 83904952a5a541ecac508449e23e70dd P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:26:57.049636 3824 consensus_queue.cc:1035] T 83904952a5a541ecac508449e23e70dd P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:26:57.051249 2094 raft_consensus.cc:2932] T 83904952a5a541ecac508449e23e70dd P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:57.055269 31387 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:57.057562 31389 catalog_manager.cc:5576] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863) does not have the latest schema for tablet fe7e65dd33b74bd4aaad54a5cf6a5626 (table rebalancer_dynamic_table [id=df5c87578fa74cf98ecf89abaeb0acfa]). Expected version 1 got 0
I20240502 21:26:57.066572 1813 tablet_service.cc:1467] Processing CreateTablet for tablet 5ddffc460e0f42e4afc2de8e6b4d3433 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=1d01050edfe94cb68d4a9cead6309bd1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:57.067018 1813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ddffc460e0f42e4afc2de8e6b4d3433. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:57.067063 31905 tablet_service.cc:1467] Processing CreateTablet for tablet f8c1b589eead429089f20b013d4e5a0f (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=1d01050edfe94cb68d4a9cead6309bd1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:57.067482 31905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8c1b589eead429089f20b013d4e5a0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:57.072602 31514 tablet_service.cc:1467] Processing CreateTablet for tablet f8c1b589eead429089f20b013d4e5a0f (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=1d01050edfe94cb68d4a9cead6309bd1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:57.072973 31514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8c1b589eead429089f20b013d4e5a0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:57.077039 4469 ts_tablet_manager.cc:1888] T 182b3f15c06c4c07ade0a8854d3ecafd P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:26:57.077211 4469 log.cc:1196] T 182b3f15c06c4c07ade0a8854d3ecafd P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/182b3f15c06c4c07ade0a8854d3ecafd
I20240502 21:26:57.077246 2208 tablet_service.cc:1467] Processing CreateTablet for tablet 5ddffc460e0f42e4afc2de8e6b4d3433 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=1d01050edfe94cb68d4a9cead6309bd1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:57.077355 31775 tablet_service.cc:1467] Processing CreateTablet for tablet e318adedad5a46ae9bca943b3cf9d818 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=1d01050edfe94cb68d4a9cead6309bd1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:57.077775 31775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e318adedad5a46ae9bca943b3cf9d818. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:57.077805 2208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ddffc460e0f42e4afc2de8e6b4d3433. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:57.080523 31516 tablet_service.cc:1467] Processing CreateTablet for tablet 5ddffc460e0f42e4afc2de8e6b4d3433 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=1d01050edfe94cb68d4a9cead6309bd1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:57.080830 1814 tablet_service.cc:1467] Processing CreateTablet for tablet f8c1b589eead429089f20b013d4e5a0f (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=1d01050edfe94cb68d4a9cead6309bd1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:57.081035 31516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ddffc460e0f42e4afc2de8e6b4d3433. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:57.080998 2210 tablet_service.cc:1467] Processing CreateTablet for tablet e318adedad5a46ae9bca943b3cf9d818 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=1d01050edfe94cb68d4a9cead6309bd1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:57.081251 1814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8c1b589eead429089f20b013d4e5a0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:57.081398 2210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e318adedad5a46ae9bca943b3cf9d818. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:57.084796 1812 tablet_service.cc:1467] Processing CreateTablet for tablet e318adedad5a46ae9bca943b3cf9d818 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=1d01050edfe94cb68d4a9cead6309bd1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:57.085199 1812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e318adedad5a46ae9bca943b3cf9d818. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:57.095376 4501 tablet.cc:1722] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 08d7a0680d244b55b93d17f9ab57b84c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:57.099913 31389 catalog_manager.cc:5576] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807) does not have the latest schema for tablet fe7e65dd33b74bd4aaad54a5cf6a5626 (table rebalancer_dynamic_table [id=df5c87578fa74cf98ecf89abaeb0acfa]). Expected version 1 got 0
I20240502 21:26:57.103236 31392 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=df5c87578fa74cf98ecf89abaeb0acfa] alter complete (version 1)
I20240502 21:26:57.105712 4505 tablet_bootstrap.cc:492] T e318adedad5a46ae9bca943b3cf9d818 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:57.106948 4501 tablet.cc:1620] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 08d7a0680d244b55b93d17f9ab57b84c: MemRowSet was empty: no flush needed.
I20240502 21:26:57.108201 4503 tablet_bootstrap.cc:492] T f8c1b589eead429089f20b013d4e5a0f P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:57.108913 4502 tablet_bootstrap.cc:492] T 5ddffc460e0f42e4afc2de8e6b4d3433 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:57.113566 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet 182b3f15c06c4c07ade0a8854d3ecafd (table rebalance_test_table_3 [id=59305f5894b34d0085a6cada5a676ca4]) successfully deleted
I20240502 21:26:57.120119 4506 tablet_bootstrap.cc:492] T e318adedad5a46ae9bca943b3cf9d818 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:57.122221 4505 tablet_bootstrap.cc:654] T e318adedad5a46ae9bca943b3cf9d818 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:57.131345 4506 tablet_bootstrap.cc:654] T e318adedad5a46ae9bca943b3cf9d818 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:57.134135 4503 tablet_bootstrap.cc:654] T f8c1b589eead429089f20b013d4e5a0f P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:57.135433 4504 tablet_bootstrap.cc:492] T f8c1b589eead429089f20b013d4e5a0f P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:57.136605 4504 tablet_bootstrap.cc:654] T f8c1b589eead429089f20b013d4e5a0f P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:57.141929 4502 tablet_bootstrap.cc:654] T 5ddffc460e0f42e4afc2de8e6b4d3433 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:57.152881 4505 tablet_bootstrap.cc:492] T e318adedad5a46ae9bca943b3cf9d818 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:57.153064 4505 ts_tablet_manager.cc:1367] T e318adedad5a46ae9bca943b3cf9d818 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.048s user 0.004s sys 0.000s
I20240502 21:26:57.153822 4505 raft_consensus.cc:348] T e318adedad5a46ae9bca943b3cf9d818 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:57.154100 4505 raft_consensus.cc:374] T e318adedad5a46ae9bca943b3cf9d818 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:57.154239 4505 raft_consensus.cc:729] T e318adedad5a46ae9bca943b3cf9d818 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:57.154520 4505 consensus_queue.cc:260] T e318adedad5a46ae9bca943b3cf9d818 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:57.156579 4505 ts_tablet_manager.cc:1398] T e318adedad5a46ae9bca943b3cf9d818 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:57.173120 4504 tablet_bootstrap.cc:492] T f8c1b589eead429089f20b013d4e5a0f P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:57.173292 4504 ts_tablet_manager.cc:1367] T f8c1b589eead429089f20b013d4e5a0f P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.000s
I20240502 21:26:57.173893 4504 raft_consensus.cc:348] T f8c1b589eead429089f20b013d4e5a0f P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:57.174134 4504 raft_consensus.cc:374] T f8c1b589eead429089f20b013d4e5a0f P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:57.174230 4504 raft_consensus.cc:729] T f8c1b589eead429089f20b013d4e5a0f P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:57.174489 4504 consensus_queue.cc:260] T f8c1b589eead429089f20b013d4e5a0f P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:57.174732 3911 raft_consensus.cc:482] T f8c1b589eead429089f20b013d4e5a0f P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:57.174865 3911 raft_consensus.cc:504] T f8c1b589eead429089f20b013d4e5a0f P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:57.175561 3911 leader_election.cc:288] T f8c1b589eead429089f20b013d4e5a0f P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:26:57.175988 4503 tablet_bootstrap.cc:492] T f8c1b589eead429089f20b013d4e5a0f P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:57.176127 4503 ts_tablet_manager.cc:1367] T f8c1b589eead429089f20b013d4e5a0f P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.068s user 0.003s sys 0.000s
I20240502 21:26:57.176842 1832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c1b589eead429089f20b013d4e5a0f" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:26:57.176910 4503 raft_consensus.cc:348] T f8c1b589eead429089f20b013d4e5a0f P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:57.177091 4503 raft_consensus.cc:374] T f8c1b589eead429089f20b013d4e5a0f P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:57.177181 4503 raft_consensus.cc:729] T f8c1b589eead429089f20b013d4e5a0f P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:57.177248 31923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c1b589eead429089f20b013d4e5a0f" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
W20240502 21:26:57.177405 31471 leader_election.cc:341] T f8c1b589eead429089f20b013d4e5a0f P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:57.177417 4503 consensus_queue.cc:260] T f8c1b589eead429089f20b013d4e5a0f P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:57.177628 31923 raft_consensus.cc:2454] T f8c1b589eead429089f20b013d4e5a0f P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:26:57.177989 31472 leader_election.cc:302] T f8c1b589eead429089f20b013d4e5a0f P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, a44f41b0b6644fb6aa29476aed8c964f; no voters: 1e04157b98dd4fad98858c290b73c3db
I20240502 21:26:57.178321 3911 raft_consensus.cc:2781] T f8c1b589eead429089f20b013d4e5a0f P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:57.178409 3911 raft_consensus.cc:482] T f8c1b589eead429089f20b013d4e5a0f P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:57.178457 3911 raft_consensus.cc:3037] T f8c1b589eead429089f20b013d4e5a0f P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:57.180687 3911 raft_consensus.cc:504] T f8c1b589eead429089f20b013d4e5a0f P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:57.181450 4167 raft_consensus.cc:482] T e318adedad5a46ae9bca943b3cf9d818 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:57.181600 4167 raft_consensus.cc:504] T e318adedad5a46ae9bca943b3cf9d818 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:57.182176 4167 leader_election.cc:288] T e318adedad5a46ae9bca943b3cf9d818 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:57.182678 31923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c1b589eead429089f20b013d4e5a0f" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:57.182878 31923 raft_consensus.cc:3037] T f8c1b589eead429089f20b013d4e5a0f P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:57.183045 2226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e318adedad5a46ae9bca943b3cf9d818" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
W20240502 21:26:57.183815 31732 leader_election.cc:341] T e318adedad5a46ae9bca943b3cf9d818 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:57.184170 1832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e318adedad5a46ae9bca943b3cf9d818" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
W20240502 21:26:57.184676 31732 leader_election.cc:341] T e318adedad5a46ae9bca943b3cf9d818 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:57.184782 31732 leader_election.cc:302] T e318adedad5a46ae9bca943b3cf9d818 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c; no voters: 1e04157b98dd4fad98858c290b73c3db, d1406e92aec84ac585304e57bd37a40d
I20240502 21:26:57.185345 3946 raft_consensus.cc:2726] T e318adedad5a46ae9bca943b3cf9d818 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:26:57.185617 31923 raft_consensus.cc:2454] T f8c1b589eead429089f20b013d4e5a0f P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:26:57.187029 31472 leader_election.cc:302] T f8c1b589eead429089f20b013d4e5a0f P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:57.187266 4188 raft_consensus.cc:2781] T f8c1b589eead429089f20b013d4e5a0f P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:57.187382 4188 raft_consensus.cc:686] T f8c1b589eead429089f20b013d4e5a0f P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:26:57.187572 4188 consensus_queue.cc:237] T f8c1b589eead429089f20b013d4e5a0f P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:57.189831 1832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c1b589eead429089f20b013d4e5a0f" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
W20240502 21:26:57.190454 31471 leader_election.cc:341] T f8c1b589eead429089f20b013d4e5a0f P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:57.190711 3911 leader_election.cc:288] T f8c1b589eead429089f20b013d4e5a0f P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Requested vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:26:57.197892 4502 tablet_bootstrap.cc:492] T 5ddffc460e0f42e4afc2de8e6b4d3433 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:57.200429 4506 tablet_bootstrap.cc:492] T e318adedad5a46ae9bca943b3cf9d818 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:57.200635 4506 ts_tablet_manager.cc:1367] T e318adedad5a46ae9bca943b3cf9d818 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.081s user 0.004s sys 0.013s
I20240502 21:26:57.201867 4502 ts_tablet_manager.cc:1367] T 5ddffc460e0f42e4afc2de8e6b4d3433 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.093s user 0.003s sys 0.000s
I20240502 21:26:57.202785 4503 ts_tablet_manager.cc:1398] T f8c1b589eead429089f20b013d4e5a0f P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20240502 21:26:57.202761 4502 raft_consensus.cc:348] T 5ddffc460e0f42e4afc2de8e6b4d3433 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:57.203009 4502 raft_consensus.cc:374] T 5ddffc460e0f42e4afc2de8e6b4d3433 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:57.203151 4502 raft_consensus.cc:729] T 5ddffc460e0f42e4afc2de8e6b4d3433 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:57.203456 4502 consensus_queue.cc:260] T 5ddffc460e0f42e4afc2de8e6b4d3433 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:57.204080 4502 ts_tablet_manager.cc:1398] T 5ddffc460e0f42e4afc2de8e6b4d3433 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:26:57.205816 4502 tablet_bootstrap.cc:492] T e318adedad5a46ae9bca943b3cf9d818 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:57.207456 4502 tablet_bootstrap.cc:654] T e318adedad5a46ae9bca943b3cf9d818 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:57.210641 4504 ts_tablet_manager.cc:1398] T f8c1b589eead429089f20b013d4e5a0f P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.037s user 0.002s sys 0.000s
I20240502 21:26:57.210932 4504 tablet_bootstrap.cc:492] T 5ddffc460e0f42e4afc2de8e6b4d3433 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:57.211582 31391 catalog_manager.cc:5484] T f8c1b589eead429089f20b013d4e5a0f P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 0 to 1, leader changed from <none> to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:57.212410 4504 tablet_bootstrap.cc:654] T 5ddffc460e0f42e4afc2de8e6b4d3433 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:57.201512 4506 raft_consensus.cc:348] T e318adedad5a46ae9bca943b3cf9d818 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:57.212911 4506 raft_consensus.cc:374] T e318adedad5a46ae9bca943b3cf9d818 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:57.213052 4506 raft_consensus.cc:729] T e318adedad5a46ae9bca943b3cf9d818 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:57.213358 4506 consensus_queue.cc:260] T e318adedad5a46ae9bca943b3cf9d818 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:57.222674 31391 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:26:57.224459 4502 tablet_bootstrap.cc:492] T e318adedad5a46ae9bca943b3cf9d818 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:57.224617 4502 ts_tablet_manager.cc:1367] T e318adedad5a46ae9bca943b3cf9d818 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.003s
I20240502 21:26:57.224720 3529 raft_consensus.cc:482] T 5ddffc460e0f42e4afc2de8e6b4d3433 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:57.224893 3529 raft_consensus.cc:504] T 5ddffc460e0f42e4afc2de8e6b4d3433 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:57.225489 4502 raft_consensus.cc:348] T e318adedad5a46ae9bca943b3cf9d818 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:57.225596 3529 leader_election.cc:288] T 5ddffc460e0f42e4afc2de8e6b4d3433 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:57.225813 4502 raft_consensus.cc:374] T e318adedad5a46ae9bca943b3cf9d818 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:57.225915 4502 raft_consensus.cc:729] T e318adedad5a46ae9bca943b3cf9d818 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:57.226119 2226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddffc460e0f42e4afc2de8e6b4d3433" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:26:57.226217 4502 consensus_queue.cc:260] T e318adedad5a46ae9bca943b3cf9d818 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:57.226604 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddffc460e0f42e4afc2de8e6b4d3433" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
W20240502 21:26:57.231765 1769 leader_election.cc:341] T 5ddffc460e0f42e4afc2de8e6b4d3433 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:26:57.232350 1770 leader_election.cc:341] T 5ddffc460e0f42e4afc2de8e6b4d3433 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:57.232504 1770 leader_election.cc:302] T 5ddffc460e0f42e4afc2de8e6b4d3433 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db; no voters: a44f41b0b6644fb6aa29476aed8c964f, d1406e92aec84ac585304e57bd37a40d
I20240502 21:26:57.233801 4502 ts_tablet_manager.cc:1398] T e318adedad5a46ae9bca943b3cf9d818 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240502 21:26:57.233793 4496 tablet.cc:1722] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:57.234277 4496 tablet.cc:1620] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b: MemRowSet was empty: no flush needed.
I20240502 21:26:57.235486 4497 tablet.cc:1722] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:57.235709 4502 tablet_bootstrap.cc:492] T f8c1b589eead429089f20b013d4e5a0f P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:57.235894 4497 tablet.cc:1620] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1e04157b98dd4fad98858c290b73c3db: MemRowSet was empty: no flush needed.
I20240502 21:26:57.237514 4336 raft_consensus.cc:2726] T 5ddffc460e0f42e4afc2de8e6b4d3433 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:26:57.238473 4506 ts_tablet_manager.cc:1398] T e318adedad5a46ae9bca943b3cf9d818 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.038s user 0.000s sys 0.002s
I20240502 21:26:57.241493 4506 tablet_bootstrap.cc:492] T 5ddffc460e0f42e4afc2de8e6b4d3433 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:57.241819 4502 tablet_bootstrap.cc:654] T f8c1b589eead429089f20b013d4e5a0f P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:57.243041 4506 tablet_bootstrap.cc:654] T 5ddffc460e0f42e4afc2de8e6b4d3433 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:57.243242 4504 tablet_bootstrap.cc:492] T 5ddffc460e0f42e4afc2de8e6b4d3433 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:26:57.243366 4504 ts_tablet_manager.cc:1367] T 5ddffc460e0f42e4afc2de8e6b4d3433 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.000s
I20240502 21:26:57.243921 4504 raft_consensus.cc:348] T 5ddffc460e0f42e4afc2de8e6b4d3433 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:57.244194 4504 raft_consensus.cc:374] T 5ddffc460e0f42e4afc2de8e6b4d3433 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:57.244261 4504 raft_consensus.cc:729] T 5ddffc460e0f42e4afc2de8e6b4d3433 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:26:57.244589 4504 consensus_queue.cc:260] T 5ddffc460e0f42e4afc2de8e6b4d3433 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:57.245136 4504 ts_tablet_manager.cc:1398] T 5ddffc460e0f42e4afc2de8e6b4d3433 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:26:57.255503 4501 tablet.cc:1722] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 08d7a0680d244b55b93d17f9ab57b84c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:57.255908 4501 tablet.cc:1620] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 08d7a0680d244b55b93d17f9ab57b84c: MemRowSet was empty: no flush needed.
I20240502 21:26:57.257812 4502 tablet_bootstrap.cc:492] T f8c1b589eead429089f20b013d4e5a0f P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:57.257930 4502 ts_tablet_manager.cc:1367] T f8c1b589eead429089f20b013d4e5a0f P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.004s
I20240502 21:26:57.258554 4502 raft_consensus.cc:348] T f8c1b589eead429089f20b013d4e5a0f P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:57.258714 4502 raft_consensus.cc:374] T f8c1b589eead429089f20b013d4e5a0f P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:57.258785 4502 raft_consensus.cc:729] T f8c1b589eead429089f20b013d4e5a0f P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:57.258951 4502 consensus_queue.cc:260] T f8c1b589eead429089f20b013d4e5a0f P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:57.259433 4502 ts_tablet_manager.cc:1398] T f8c1b589eead429089f20b013d4e5a0f P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240502 21:26:57.262133 4506 tablet_bootstrap.cc:492] T 5ddffc460e0f42e4afc2de8e6b4d3433 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:57.262295 4506 ts_tablet_manager.cc:1367] T 5ddffc460e0f42e4afc2de8e6b4d3433 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.002s
I20240502 21:26:57.266120 4506 raft_consensus.cc:348] T 5ddffc460e0f42e4afc2de8e6b4d3433 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:57.266302 4506 raft_consensus.cc:374] T 5ddffc460e0f42e4afc2de8e6b4d3433 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:57.266367 4506 raft_consensus.cc:729] T 5ddffc460e0f42e4afc2de8e6b4d3433 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:57.268697 4506 consensus_queue.cc:260] T 5ddffc460e0f42e4afc2de8e6b4d3433 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:57.259964 31386 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=df5c87578fa74cf98ecf89abaeb0acfa] alter complete (version 2)
I20240502 21:26:57.271292 4506 ts_tablet_manager.cc:1398] T 5ddffc460e0f42e4afc2de8e6b4d3433 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240502 21:26:57.303853 4499 raft_consensus.cc:482] T 5ddffc460e0f42e4afc2de8e6b4d3433 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:57.304018 4499 raft_consensus.cc:504] T 5ddffc460e0f42e4afc2de8e6b4d3433 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:57.305562 4499 leader_election.cc:288] T 5ddffc460e0f42e4afc2de8e6b4d3433 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:57.305867 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddffc460e0f42e4afc2de8e6b4d3433" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:26:57.306051 31536 raft_consensus.cc:2454] T 5ddffc460e0f42e4afc2de8e6b4d3433 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:26:57.306365 1832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddffc460e0f42e4afc2de8e6b4d3433" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:26:57.306447 2164 leader_election.cc:302] T 5ddffc460e0f42e4afc2de8e6b4d3433 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a44f41b0b6644fb6aa29476aed8c964f, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:57.306607 1832 raft_consensus.cc:2454] T 5ddffc460e0f42e4afc2de8e6b4d3433 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:26:57.307348 4499 raft_consensus.cc:2781] T 5ddffc460e0f42e4afc2de8e6b4d3433 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:57.307467 4499 raft_consensus.cc:482] T 5ddffc460e0f42e4afc2de8e6b4d3433 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:57.307559 4499 raft_consensus.cc:3037] T 5ddffc460e0f42e4afc2de8e6b4d3433 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:57.311012 4499 raft_consensus.cc:504] T 5ddffc460e0f42e4afc2de8e6b4d3433 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:57.312098 1832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddffc460e0f42e4afc2de8e6b4d3433" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:26:57.312296 1832 raft_consensus.cc:3037] T 5ddffc460e0f42e4afc2de8e6b4d3433 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:57.315644 1832 raft_consensus.cc:2454] T 5ddffc460e0f42e4afc2de8e6b4d3433 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 1.
I20240502 21:26:57.316260 2163 leader_election.cc:302] T 5ddffc460e0f42e4afc2de8e6b4d3433 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:57.316563 4498 raft_consensus.cc:2781] T 5ddffc460e0f42e4afc2de8e6b4d3433 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:57.316704 4498 raft_consensus.cc:686] T 5ddffc460e0f42e4afc2de8e6b4d3433 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Becoming Leader. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Running, Role: LEADER
I20240502 21:26:57.317005 4498 consensus_queue.cc:237] T 5ddffc460e0f42e4afc2de8e6b4d3433 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:26:57.320969 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddffc460e0f42e4afc2de8e6b4d3433" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:26:57.321169 31536 raft_consensus.cc:3037] T 5ddffc460e0f42e4afc2de8e6b4d3433 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:57.324090 31536 raft_consensus.cc:2454] T 5ddffc460e0f42e4afc2de8e6b4d3433 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 1.
I20240502 21:26:57.326006 4499 leader_election.cc:288] T 5ddffc460e0f42e4afc2de8e6b4d3433 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Requested vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:57.350531 31389 catalog_manager.cc:5484] T 5ddffc460e0f42e4afc2de8e6b4d3433 P d1406e92aec84ac585304e57bd37a40d reported cstate change: term changed from 0 to 1, leader changed from <none> to d1406e92aec84ac585304e57bd37a40d (127.24.226.136). New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:57.370864 31389 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:26:57.382377 4496 tablet.cc:1722] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:57.383021 4497 tablet.cc:1722] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:57.393424 4501 tablet.cc:1722] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 08d7a0680d244b55b93d17f9ab57b84c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:57.398183 31392 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table_new_name [id=df5c87578fa74cf98ecf89abaeb0acfa] alter complete (version 3)
I20240502 21:26:57.434557 1832 consensus_queue.cc:237] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } }
I20240502 21:26:57.436990 2094 raft_consensus.cc:1261] T 2c737c49a32b4f3ea5ef735fa254d48e P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:57.437005 31795 raft_consensus.cc:1261] T 2c737c49a32b4f3ea5ef735fa254d48e P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:57.437685 4336 consensus_queue.cc:1035] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:26:57.438025 1770 consensus_peers.cc:485] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db -> Peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): Couldn't send request to peer 05f4fac41c4f4959b9b316cb112d0ada. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c737c49a32b4f3ea5ef735fa254d48e. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:57.438411 3529 consensus_queue.cc:1035] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:57.444463 3529 raft_consensus.cc:2932] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } } }
I20240502 21:26:57.448304 31795 raft_consensus.cc:2932] T 2c737c49a32b4f3ea5ef735fa254d48e P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } } }
I20240502 21:26:57.450938 2097 raft_consensus.cc:2932] T 2c737c49a32b4f3ea5ef735fa254d48e P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } } }
I20240502 21:26:57.457104 31389 catalog_manager.cc:5484] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db reported cstate change: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:57.473313 2226 consensus_queue.cc:237] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } }
W20240502 21:26:57.476152 2164 consensus_peers.cc:485] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d -> Peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Couldn't send request to peer a44f41b0b6644fb6aa29476aed8c964f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2068521c26dc44f9ba60e274eb039b22. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:57.476390 1964 raft_consensus.cc:1261] T 2068521c26dc44f9ba60e274eb039b22 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:57.477438 4407 consensus_queue.cc:1035] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:57.480587 2094 raft_consensus.cc:1261] T 2068521c26dc44f9ba60e274eb039b22 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:57.481175 4243 consensus_queue.cc:1035] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:57.481174 4499 raft_consensus.cc:2932] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } } }
I20240502 21:26:57.481618 1964 raft_consensus.cc:2932] T 2068521c26dc44f9ba60e274eb039b22 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } } }
I20240502 21:26:57.490289 2094 raft_consensus.cc:2932] T 2068521c26dc44f9ba60e274eb039b22 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } } }
I20240502 21:26:57.492313 31386 catalog_manager.cc:5484] T 2068521c26dc44f9ba60e274eb039b22 P 9dec645b473f40578e726f3e50e219bf reported cstate change: config changed from index -1 to 2, NON_VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) added. New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } } }
I20240502 21:26:57.502946 31392 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:57.503170 31392 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:57.506947 31392 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet fe7e65dd33b74bd4aaad54a5cf6a5626
I20240502 21:26:57.508194 1814 tablet_service.cc:1514] Processing DeleteTablet for tablet fe7e65dd33b74bd4aaad54a5cf6a5626 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:57 UTC) from {username='slave'} at 127.0.0.1:47510
I20240502 21:26:57.508649 31775 tablet_service.cc:1514] Processing DeleteTablet for tablet fe7e65dd33b74bd4aaad54a5cf6a5626 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:57 UTC) from {username='slave'} at 127.0.0.1:51596
I20240502 21:26:57.508966 4456 tablet_replica.cc:330] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 08d7a0680d244b55b93d17f9ab57b84c: stopping tablet replica
I20240502 21:26:57.508986 2076 tablet_service.cc:1514] Processing DeleteTablet for tablet fe7e65dd33b74bd4aaad54a5cf6a5626 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:57 UTC) from {username='slave'} at 127.0.0.1:36740
I20240502 21:26:57.509153 4456 raft_consensus.cc:2229] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:57.509387 4456 raft_consensus.cc:2258] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:57.510105 4540 tablet_replica.cc:330] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b: stopping tablet replica
I20240502 21:26:57.510339 4540 raft_consensus.cc:2229] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:57.510653 4540 raft_consensus.cc:2258] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:57.511548 4540 ts_tablet_manager.cc:1875] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:57.511723 4456 ts_tablet_manager.cc:1875] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:57.517890 4540 ts_tablet_manager.cc:1888] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:57.518100 4540 log.cc:1196] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal/wals/fe7e65dd33b74bd4aaad54a5cf6a5626
I20240502 21:26:57.518613 4540 ts_tablet_manager.cc:1909] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting consensus metadata
I20240502 21:26:57.519587 31381 catalog_manager.cc:4830] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): tablet fe7e65dd33b74bd4aaad54a5cf6a5626 (table rebalancer_dynamic_table_new_name [id=df5c87578fa74cf98ecf89abaeb0acfa]) successfully deleted
I20240502 21:26:57.522243 4456 ts_tablet_manager.cc:1888] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 08d7a0680d244b55b93d17f9ab57b84c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:57.522447 4456 log.cc:1196] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/wals/fe7e65dd33b74bd4aaad54a5cf6a5626
I20240502 21:26:57.524396 4456 ts_tablet_manager.cc:1909] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting consensus metadata
I20240502 21:26:57.525251 4539 tablet_replica.cc:330] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1e04157b98dd4fad98858c290b73c3db: stopping tablet replica
I20240502 21:26:57.526994 4539 raft_consensus.cc:2229] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:57.526991 31381 catalog_manager.cc:4830] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): tablet fe7e65dd33b74bd4aaad54a5cf6a5626 (table rebalancer_dynamic_table_new_name [id=df5c87578fa74cf98ecf89abaeb0acfa]) successfully deleted
I20240502 21:26:57.527345 4539 raft_consensus.cc:2258] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:57.528151 4539 ts_tablet_manager.cc:1875] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1e04157b98dd4fad98858c290b73c3db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:57.534265 4539 ts_tablet_manager.cc:1888] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1e04157b98dd4fad98858c290b73c3db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:57.534451 4539 log.cc:1196] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1e04157b98dd4fad98858c290b73c3db: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal/wals/fe7e65dd33b74bd4aaad54a5cf6a5626
I20240502 21:26:57.535116 4539 ts_tablet_manager.cc:1909] T fe7e65dd33b74bd4aaad54a5cf6a5626 P 1e04157b98dd4fad98858c290b73c3db: Deleting consensus metadata
I20240502 21:26:57.536065 31381 catalog_manager.cc:4830] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): tablet fe7e65dd33b74bd4aaad54a5cf6a5626 (table rebalancer_dynamic_table_new_name [id=df5c87578fa74cf98ecf89abaeb0acfa]) successfully deleted
I20240502 21:26:57.539165 2226 consensus_queue.cc:237] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } }
I20240502 21:26:57.541134 4526 raft_consensus.cc:482] T e318adedad5a46ae9bca943b3cf9d818 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:57.541267 4526 raft_consensus.cc:504] T e318adedad5a46ae9bca943b3cf9d818 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:57.541411 1964 raft_consensus.cc:1261] T 0d923caa3f034a899050fe62dfc0c3d8 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:57.541935 4526 leader_election.cc:288] T e318adedad5a46ae9bca943b3cf9d818 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:57.541888 4498 consensus_queue.cc:1035] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:57.542111 2094 raft_consensus.cc:1261] T 0d923caa3f034a899050fe62dfc0c3d8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:57.542729 2226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e318adedad5a46ae9bca943b3cf9d818" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:26:57.542914 2226 raft_consensus.cc:2454] T e318adedad5a46ae9bca943b3cf9d818 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 0.
I20240502 21:26:57.543089 31795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e318adedad5a46ae9bca943b3cf9d818" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:57.543218 1769 leader_election.cc:302] T e318adedad5a46ae9bca943b3cf9d818 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:57.543306 31795 raft_consensus.cc:2454] T e318adedad5a46ae9bca943b3cf9d818 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 0.
I20240502 21:26:57.543428 4526 raft_consensus.cc:2781] T e318adedad5a46ae9bca943b3cf9d818 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:57.543510 4526 raft_consensus.cc:482] T e318adedad5a46ae9bca943b3cf9d818 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:57.543557 4526 raft_consensus.cc:3037] T e318adedad5a46ae9bca943b3cf9d818 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:26:57.544987 2163 consensus_peers.cc:485] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d -> Peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): Couldn't send request to peer 08d7a0680d244b55b93d17f9ab57b84c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d923caa3f034a899050fe62dfc0c3d8. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:57.545338 4498 consensus_queue.cc:1035] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:57.545616 4526 raft_consensus.cc:504] T e318adedad5a46ae9bca943b3cf9d818 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:57.546105 4526 leader_election.cc:288] T e318adedad5a46ae9bca943b3cf9d818 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Requested vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:57.546522 31795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e318adedad5a46ae9bca943b3cf9d818" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:57.546685 31795 raft_consensus.cc:3037] T e318adedad5a46ae9bca943b3cf9d818 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:57.549798 4498 raft_consensus.cc:2932] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } } }
I20240502 21:26:57.550122 2230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e318adedad5a46ae9bca943b3cf9d818" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:26:57.550350 2230 raft_consensus.cc:3037] T e318adedad5a46ae9bca943b3cf9d818 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:57.553505 2230 raft_consensus.cc:2454] T e318adedad5a46ae9bca943b3cf9d818 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:26:57.553907 1769 leader_election.cc:302] T e318adedad5a46ae9bca943b3cf9d818 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:57.554227 4526 raft_consensus.cc:2781] T e318adedad5a46ae9bca943b3cf9d818 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:57.554379 1964 raft_consensus.cc:2932] T 0d923caa3f034a899050fe62dfc0c3d8 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } } }
I20240502 21:26:57.554718 4526 raft_consensus.cc:686] T e318adedad5a46ae9bca943b3cf9d818 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Becoming Leader. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Running, Role: LEADER
I20240502 21:26:57.555078 4526 consensus_queue.cc:237] T e318adedad5a46ae9bca943b3cf9d818 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:57.555835 2094 raft_consensus.cc:2932] T 0d923caa3f034a899050fe62dfc0c3d8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } } }
I20240502 21:26:57.548420 31795 raft_consensus.cc:2454] T e318adedad5a46ae9bca943b3cf9d818 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:26:57.559021 31392 catalog_manager.cc:5484] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d reported cstate change: config changed from index -1 to 2, NON_VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) added. New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:57.568012 1832 raft_consensus.cc:3037] T f8c1b589eead429089f20b013d4e5a0f P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:57.572602 4430 consensus_queue.cc:1035] T f8c1b589eead429089f20b013d4e5a0f P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:57.581902 3911 consensus_queue.cc:1035] T f8c1b589eead429089f20b013d4e5a0f P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:57.599817 31389 catalog_manager.cc:5484] T e318adedad5a46ae9bca943b3cf9d818 P 1e04157b98dd4fad98858c290b73c3db reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e04157b98dd4fad98858c290b73c3db (127.24.226.133). New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:57.611353 31389 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:26:57.616703 31795 consensus_queue.cc:237] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } }
I20240502 21:26:57.619834 2094 raft_consensus.cc:1261] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:57.620942 4167 consensus_queue.cc:1035] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:57.621698 2230 raft_consensus.cc:1261] T 534b4d96df9b4bc5a9480b6c45eb11a5 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:57.623785 2076 tablet_service.cc:1467] Processing CreateTablet for tablet 929492106d2a4fdaad13a82b8ff36f7b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cb776fab0f3e40559b48fb13dbded7d9]), partition=
I20240502 21:26:57.623764 2210 tablet_service.cc:1467] Processing CreateTablet for tablet 929492106d2a4fdaad13a82b8ff36f7b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cb776fab0f3e40559b48fb13dbded7d9]), partition=
I20240502 21:26:57.623965 1946 tablet_service.cc:1467] Processing CreateTablet for tablet 929492106d2a4fdaad13a82b8ff36f7b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cb776fab0f3e40559b48fb13dbded7d9]), partition=
I20240502 21:26:57.624117 2076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 929492106d2a4fdaad13a82b8ff36f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:57.624140 2210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 929492106d2a4fdaad13a82b8ff36f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:57.624681 4167 consensus_queue.cc:1035] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:57.624431 1946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 929492106d2a4fdaad13a82b8ff36f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:57.630120 4506 tablet_bootstrap.cc:492] T 929492106d2a4fdaad13a82b8ff36f7b P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:57.631860 4506 tablet_bootstrap.cc:654] T 929492106d2a4fdaad13a82b8ff36f7b P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
W20240502 21:26:57.637378 31733 consensus_peers.cc:485] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c -> Peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): Couldn't send request to peer 2d50f33557944bb480ea32199d9ee4f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 534b4d96df9b4bc5a9480b6c45eb11a5. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:57.639571 4167 raft_consensus.cc:2932] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } } }
I20240502 21:26:57.640980 2094 raft_consensus.cc:2932] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } } }
I20240502 21:26:57.646626 4563 tablet_bootstrap.cc:492] T 929492106d2a4fdaad13a82b8ff36f7b P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:57.649744 4506 tablet_bootstrap.cc:492] T 929492106d2a4fdaad13a82b8ff36f7b P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:57.649933 4506 ts_tablet_manager.cc:1367] T 929492106d2a4fdaad13a82b8ff36f7b P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.005s
I20240502 21:26:57.649971 2226 raft_consensus.cc:2932] T 534b4d96df9b4bc5a9480b6c45eb11a5 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } } }
I20240502 21:26:57.650678 4506 raft_consensus.cc:348] T 929492106d2a4fdaad13a82b8ff36f7b P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:57.650867 4506 raft_consensus.cc:374] T 929492106d2a4fdaad13a82b8ff36f7b P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:57.651005 4506 raft_consensus.cc:729] T 929492106d2a4fdaad13a82b8ff36f7b P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:57.651280 4506 consensus_queue.cc:260] T 929492106d2a4fdaad13a82b8ff36f7b P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:57.652460 31389 catalog_manager.cc:5484] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index -1 to 2, NON_VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) added. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:57.654673 4564 tablet_bootstrap.cc:492] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:57.657436 4564 tablet_bootstrap.cc:654] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:57.662374 4563 tablet_bootstrap.cc:654] T 929492106d2a4fdaad13a82b8ff36f7b P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:57.663712 4506 ts_tablet_manager.cc:1398] T 929492106d2a4fdaad13a82b8ff36f7b P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240502 21:26:57.675957 4564 tablet_bootstrap.cc:492] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:26:57.676245 4564 ts_tablet_manager.cc:1367] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.007s
I20240502 21:26:57.677361 4564 raft_consensus.cc:348] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:57.677670 4564 raft_consensus.cc:374] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:57.677809 4564 raft_consensus.cc:729] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:26:57.678117 4564 consensus_queue.cc:260] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:57.679960 4564 ts_tablet_manager.cc:1398] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 21:26:57.680408 4563 tablet_bootstrap.cc:492] T 929492106d2a4fdaad13a82b8ff36f7b P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:57.680553 4563 ts_tablet_manager.cc:1367] T 929492106d2a4fdaad13a82b8ff36f7b P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.034s user 0.009s sys 0.000s
I20240502 21:26:57.681205 4563 raft_consensus.cc:348] T 929492106d2a4fdaad13a82b8ff36f7b P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:57.681489 4563 raft_consensus.cc:374] T 929492106d2a4fdaad13a82b8ff36f7b P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:57.681566 4563 raft_consensus.cc:729] T 929492106d2a4fdaad13a82b8ff36f7b P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:57.681730 4563 consensus_queue.cc:260] T 929492106d2a4fdaad13a82b8ff36f7b P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:57.682206 4563 ts_tablet_manager.cc:1398] T 929492106d2a4fdaad13a82b8ff36f7b P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:57.700753 31795 consensus_queue.cc:237] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } }
I20240502 21:26:57.702665 31536 raft_consensus.cc:1261] T 53b7eec870c44999b48f0bf744b2ffcd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:26:57.702698 2230 raft_consensus.cc:1261] T 53b7eec870c44999b48f0bf744b2ffcd P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 21:26:57.702886 31732 consensus_peers.cc:485] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c -> Peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): Couldn't send request to peer 1e04157b98dd4fad98858c290b73c3db. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 53b7eec870c44999b48f0bf744b2ffcd. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:57.703141 4167 consensus_queue.cc:1035] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:26:57.703403 4559 consensus_queue.cc:1035] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:26:57.706312 4559 raft_consensus.cc:2932] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } } }
I20240502 21:26:57.707367 2230 raft_consensus.cc:2932] T 53b7eec870c44999b48f0bf744b2ffcd P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } } }
I20240502 21:26:57.710783 31536 raft_consensus.cc:2932] T 53b7eec870c44999b48f0bf744b2ffcd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } } }
I20240502 21:26:57.712244 31389 catalog_manager.cc:5484] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index 6 to 7, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:57.729607 4498 consensus_queue.cc:1035] T 5ddffc460e0f42e4afc2de8e6b4d3433 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:57.758168 4243 consensus_queue.cc:1035] T 5ddffc460e0f42e4afc2de8e6b4d3433 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:57.781495 31795 consensus_queue.cc:237] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } }
W20240502 21:26:57.784896 31732 consensus_peers.cc:485] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c -> Peer 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): Couldn't send request to peer 1ade0354c44c4c37ae480c7a94b11e1b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0f96b56fb68439a97cf2aae4aa89b57. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:57.784924 31923 raft_consensus.cc:1261] T e0f96b56fb68439a97cf2aae4aa89b57 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:57.785441 31665 raft_consensus.cc:1261] T e0f96b56fb68439a97cf2aae4aa89b57 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:57.785949 3946 consensus_queue.cc:1035] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:57.786243 4167 consensus_queue.cc:1035] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:57.802021 4559 raft_consensus.cc:2932] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } } }
I20240502 21:26:57.802832 31666 raft_consensus.cc:2932] T e0f96b56fb68439a97cf2aae4aa89b57 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } } }
I20240502 21:26:57.805178 4581 raft_consensus.cc:482] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:57.805363 4581 raft_consensus.cc:504] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:57.806063 4581 leader_election.cc:288] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:57.806668 2230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929492106d2a4fdaad13a82b8ff36f7b" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:26:57.806877 2230 raft_consensus.cc:2454] T 929492106d2a4fdaad13a82b8ff36f7b P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 0.
I20240502 21:26:57.807221 2094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929492106d2a4fdaad13a82b8ff36f7b" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:57.807423 2094 raft_consensus.cc:2454] T 929492106d2a4fdaad13a82b8ff36f7b P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 0.
I20240502 21:26:57.808995 1901 leader_election.cc:302] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dec645b473f40578e726f3e50e219bf, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:57.809517 4581 raft_consensus.cc:2781] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:57.809623 4581 raft_consensus.cc:482] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:57.809710 4581 raft_consensus.cc:3037] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:57.812121 4581 raft_consensus.cc:504] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:57.813550 4581 leader_election.cc:288] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 election: Requested vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:57.815807 2226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929492106d2a4fdaad13a82b8ff36f7b" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:26:57.815989 2226 raft_consensus.cc:3037] T 929492106d2a4fdaad13a82b8ff36f7b P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:57.816255 2094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929492106d2a4fdaad13a82b8ff36f7b" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:26:57.816459 2094 raft_consensus.cc:3037] T 929492106d2a4fdaad13a82b8ff36f7b P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:57.818934 2226 raft_consensus.cc:2454] T 929492106d2a4fdaad13a82b8ff36f7b P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 1.
I20240502 21:26:57.819280 2094 raft_consensus.cc:2454] T 929492106d2a4fdaad13a82b8ff36f7b P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 1.
I20240502 21:26:57.820619 1901 leader_election.cc:302] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:26:57.820839 31923 raft_consensus.cc:2932] T e0f96b56fb68439a97cf2aae4aa89b57 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } } }
I20240502 21:26:57.821192 31391 catalog_manager.cc:5484] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:57.824597 4581 raft_consensus.cc:2781] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:57.824749 4581 raft_consensus.cc:686] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf [term 1 LEADER]: Becoming Leader. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Running, Role: LEADER
I20240502 21:26:57.825043 4581 consensus_queue.cc:237] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:57.829885 31386 catalog_manager.cc:5484] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dec645b473f40578e726f3e50e219bf (127.24.226.134). New cstate: current_term: 1 leader_uuid: "9dec645b473f40578e726f3e50e219bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:57.895169 2226 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f13c3b9c4b7c471fa34156b781473ec4"
dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38252
I20240502 21:26:57.895433 2226 raft_consensus.cc:595] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Received request to transfer leadership
I20240502 21:26:57.920317 4636 ts_tablet_manager.cc:902] T 2c737c49a32b4f3ea5ef735fa254d48e P 05f4fac41c4f4959b9b316cb112d0ada: Initiating tablet copy from peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:26:57.922160 4636 tablet_copy_client.cc:308] T 2c737c49a32b4f3ea5ef735fa254d48e P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.133:39863
I20240502 21:26:57.922809 1854 tablet_copy_service.cc:140] P 1e04157b98dd4fad98858c290b73c3db: Received BeginTabletCopySession request for tablet 2c737c49a32b4f3ea5ef735fa254d48e from peer 05f4fac41c4f4959b9b316cb112d0ada ({username='slave'} at 127.24.226.130:43577)
I20240502 21:26:57.923002 1854 tablet_copy_service.cc:161] P 1e04157b98dd4fad98858c290b73c3db: Beginning new tablet copy session on tablet 2c737c49a32b4f3ea5ef735fa254d48e from peer 05f4fac41c4f4959b9b316cb112d0ada at {username='slave'} at 127.24.226.130:43577: session id = 05f4fac41c4f4959b9b316cb112d0ada-2c737c49a32b4f3ea5ef735fa254d48e
I20240502 21:26:57.926046 1854 tablet_copy_source_session.cc:205] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:57.927424 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c737c49a32b4f3ea5ef735fa254d48e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:57.932170 4636 tablet_copy_client.cc:783] T 2c737c49a32b4f3ea5ef735fa254d48e P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:57.932390 4636 tablet_copy_client.cc:647] T 2c737c49a32b4f3ea5ef735fa254d48e P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:57.933925 4636 tablet_copy_client.cc:523] T 2c737c49a32b4f3ea5ef735fa254d48e P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:57.937383 4636 tablet_bootstrap.cc:492] T 2c737c49a32b4f3ea5ef735fa254d48e P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:57.942631 4636 tablet_bootstrap.cc:492] T 2c737c49a32b4f3ea5ef735fa254d48e P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:57.943055 4636 tablet_bootstrap.cc:492] T 2c737c49a32b4f3ea5ef735fa254d48e P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap complete.
I20240502 21:26:57.943300 4636 ts_tablet_manager.cc:1367] T 2c737c49a32b4f3ea5ef735fa254d48e P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240502 21:26:57.944141 4636 raft_consensus.cc:348] T 2c737c49a32b4f3ea5ef735fa254d48e P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } }
I20240502 21:26:57.944340 4636 raft_consensus.cc:729] T 2c737c49a32b4f3ea5ef735fa254d48e P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: LEARNER
I20240502 21:26:57.944576 4636 consensus_queue.cc:260] T 2c737c49a32b4f3ea5ef735fa254d48e P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } }
I20240502 21:26:57.945243 4636 ts_tablet_manager.cc:1398] T 2c737c49a32b4f3ea5ef735fa254d48e P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:26:57.946408 1854 tablet_copy_service.cc:342] P 1e04157b98dd4fad98858c290b73c3db: Request end of tablet copy session 05f4fac41c4f4959b9b316cb112d0ada-2c737c49a32b4f3ea5ef735fa254d48e received from {username='slave'} at 127.24.226.130:43577
I20240502 21:26:57.946544 1854 tablet_copy_service.cc:433] P 1e04157b98dd4fad98858c290b73c3db: ending tablet copy session 05f4fac41c4f4959b9b316cb112d0ada-2c737c49a32b4f3ea5ef735fa254d48e on tablet 2c737c49a32b4f3ea5ef735fa254d48e with peer 05f4fac41c4f4959b9b316cb112d0ada
I20240502 21:26:57.949918 31391 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:26:57.956969 2230 raft_consensus.cc:1261] T 929492106d2a4fdaad13a82b8ff36f7b P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:57.957151 2094 raft_consensus.cc:1261] T 929492106d2a4fdaad13a82b8ff36f7b P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:57.957576 3818 consensus_queue.cc:1035] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:57.961431 4581 consensus_queue.cc:1035] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:57.962916 31386 catalog_manager.cc:5576] TS d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907) does not have the latest schema for tablet 929492106d2a4fdaad13a82b8ff36f7b (table rebalancer_dynamic_table [id=cb776fab0f3e40559b48fb13dbded7d9]). Expected version 1 got 0
I20240502 21:26:57.964696 4660 tablet.cc:1722] T 929492106d2a4fdaad13a82b8ff36f7b P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:57.965036 4659 tablet.cc:1722] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:57.965595 4660 tablet.cc:1620] T 929492106d2a4fdaad13a82b8ff36f7b P d1406e92aec84ac585304e57bd37a40d: MemRowSet was empty: no flush needed.
I20240502 21:26:57.966257 4659 tablet.cc:1620] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf: MemRowSet was empty: no flush needed.
I20240502 21:26:57.972278 31386 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=cb776fab0f3e40559b48fb13dbded7d9] alter complete (version 1)
I20240502 21:26:57.990221 31391 catalog_manager.cc:5576] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569) does not have the latest schema for tablet 929492106d2a4fdaad13a82b8ff36f7b (table rebalancer_dynamic_table [id=cb776fab0f3e40559b48fb13dbded7d9]). Expected version 1 got 0
I20240502 21:26:57.991297 4664 tablet.cc:1722] T 929492106d2a4fdaad13a82b8ff36f7b P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:57.992061 4664 tablet.cc:1620] T 929492106d2a4fdaad13a82b8ff36f7b P 1ade0354c44c4c37ae480c7a94b11e1b: MemRowSet was empty: no flush needed.
I20240502 21:26:58.011281 4666 ts_tablet_manager.cc:902] T 2068521c26dc44f9ba60e274eb039b22 P a44f41b0b6644fb6aa29476aed8c964f: Initiating tablet copy from peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:58.025513 4526 consensus_queue.cc:1035] T e318adedad5a46ae9bca943b3cf9d818 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:58.032353 4666 tablet_copy_client.cc:308] T 2068521c26dc44f9ba60e274eb039b22 P a44f41b0b6644fb6aa29476aed8c964f: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.136:34907
I20240502 21:26:58.032996 2251 tablet_copy_service.cc:140] P d1406e92aec84ac585304e57bd37a40d: Received BeginTabletCopySession request for tablet 2068521c26dc44f9ba60e274eb039b22 from peer a44f41b0b6644fb6aa29476aed8c964f ({username='slave'} at 127.24.226.129:40523)
I20240502 21:26:58.033218 2251 tablet_copy_service.cc:161] P d1406e92aec84ac585304e57bd37a40d: Beginning new tablet copy session on tablet 2068521c26dc44f9ba60e274eb039b22 from peer a44f41b0b6644fb6aa29476aed8c964f at {username='slave'} at 127.24.226.129:40523: session id = a44f41b0b6644fb6aa29476aed8c964f-2068521c26dc44f9ba60e274eb039b22
I20240502 21:26:58.036279 2251 tablet_copy_source_session.cc:205] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:58.049252 4666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2068521c26dc44f9ba60e274eb039b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:58.057085 4674 ts_tablet_manager.cc:902] T 0d923caa3f034a899050fe62dfc0c3d8 P 08d7a0680d244b55b93d17f9ab57b84c: Initiating tablet copy from peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:58.061584 4666 tablet_copy_client.cc:783] T 2068521c26dc44f9ba60e274eb039b22 P a44f41b0b6644fb6aa29476aed8c964f: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:58.061760 4666 tablet_copy_client.cc:647] T 2068521c26dc44f9ba60e274eb039b22 P a44f41b0b6644fb6aa29476aed8c964f: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:58.066735 4666 tablet_copy_client.cc:523] T 2068521c26dc44f9ba60e274eb039b22 P a44f41b0b6644fb6aa29476aed8c964f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:58.069195 4526 consensus_queue.cc:1035] T e318adedad5a46ae9bca943b3cf9d818 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:58.072396 4666 tablet_bootstrap.cc:492] T 2068521c26dc44f9ba60e274eb039b22 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:26:58.074611 31391 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:26:58.086786 4666 tablet_bootstrap.cc:492] T 2068521c26dc44f9ba60e274eb039b22 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:58.088060 4659 tablet.cc:1722] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:58.088557 4659 tablet.cc:1620] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf: MemRowSet was empty: no flush needed.
I20240502 21:26:58.089062 4664 tablet.cc:1722] T 929492106d2a4fdaad13a82b8ff36f7b P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:58.089187 4660 tablet.cc:1722] T 929492106d2a4fdaad13a82b8ff36f7b P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:58.089511 4664 tablet.cc:1620] T 929492106d2a4fdaad13a82b8ff36f7b P 1ade0354c44c4c37ae480c7a94b11e1b: MemRowSet was empty: no flush needed.
I20240502 21:26:58.089613 4660 tablet.cc:1620] T 929492106d2a4fdaad13a82b8ff36f7b P d1406e92aec84ac585304e57bd37a40d: MemRowSet was empty: no flush needed.
I20240502 21:26:58.095662 4666 tablet_bootstrap.cc:492] T 2068521c26dc44f9ba60e274eb039b22 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap complete.
I20240502 21:26:58.097493 4666 ts_tablet_manager.cc:1367] T 2068521c26dc44f9ba60e274eb039b22 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.001s
I20240502 21:26:58.098367 4666 raft_consensus.cc:348] T 2068521c26dc44f9ba60e274eb039b22 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } }
I20240502 21:26:58.099375 4666 raft_consensus.cc:729] T 2068521c26dc44f9ba60e274eb039b22 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: LEARNER
I20240502 21:26:58.099720 4666 consensus_queue.cc:260] T 2068521c26dc44f9ba60e274eb039b22 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } }
I20240502 21:26:58.100714 4666 ts_tablet_manager.cc:1398] T 2068521c26dc44f9ba60e274eb039b22 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:26:58.102164 2251 tablet_copy_service.cc:342] P d1406e92aec84ac585304e57bd37a40d: Request end of tablet copy session a44f41b0b6644fb6aa29476aed8c964f-2068521c26dc44f9ba60e274eb039b22 received from {username='slave'} at 127.24.226.129:40523
I20240502 21:26:58.102277 2251 tablet_copy_service.cc:433] P d1406e92aec84ac585304e57bd37a40d: ending tablet copy session a44f41b0b6644fb6aa29476aed8c964f-2068521c26dc44f9ba60e274eb039b22 on tablet 2068521c26dc44f9ba60e274eb039b22 with peer a44f41b0b6644fb6aa29476aed8c964f
I20240502 21:26:58.111842 31392 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=cb776fab0f3e40559b48fb13dbded7d9] alter complete (version 2)
I20240502 21:26:58.120961 4674 tablet_copy_client.cc:308] T 0d923caa3f034a899050fe62dfc0c3d8 P 08d7a0680d244b55b93d17f9ab57b84c: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.136:34907
I20240502 21:26:58.121620 2251 tablet_copy_service.cc:140] P d1406e92aec84ac585304e57bd37a40d: Received BeginTabletCopySession request for tablet 0d923caa3f034a899050fe62dfc0c3d8 from peer 08d7a0680d244b55b93d17f9ab57b84c ({username='slave'} at 127.24.226.131:53089)
I20240502 21:26:58.121784 2251 tablet_copy_service.cc:161] P d1406e92aec84ac585304e57bd37a40d: Beginning new tablet copy session on tablet 0d923caa3f034a899050fe62dfc0c3d8 from peer 08d7a0680d244b55b93d17f9ab57b84c at {username='slave'} at 127.24.226.131:53089: session id = 08d7a0680d244b55b93d17f9ab57b84c-0d923caa3f034a899050fe62dfc0c3d8
I20240502 21:26:58.125972 2251 tablet_copy_source_session.cc:205] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:58.127374 4674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d923caa3f034a899050fe62dfc0c3d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:58.133037 4674 tablet_copy_client.cc:783] T 0d923caa3f034a899050fe62dfc0c3d8 P 08d7a0680d244b55b93d17f9ab57b84c: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:58.133268 4674 tablet_copy_client.cc:647] T 0d923caa3f034a899050fe62dfc0c3d8 P 08d7a0680d244b55b93d17f9ab57b84c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:58.134929 4674 tablet_copy_client.cc:523] T 0d923caa3f034a899050fe62dfc0c3d8 P 08d7a0680d244b55b93d17f9ab57b84c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:58.137588 4674 tablet_bootstrap.cc:492] T 0d923caa3f034a899050fe62dfc0c3d8 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:58.142695 4674 tablet_bootstrap.cc:492] T 0d923caa3f034a899050fe62dfc0c3d8 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:58.143108 4674 tablet_bootstrap.cc:492] T 0d923caa3f034a899050fe62dfc0c3d8 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap complete.
I20240502 21:26:58.143370 4674 ts_tablet_manager.cc:1367] T 0d923caa3f034a899050fe62dfc0c3d8 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 21:26:58.144040 4674 raft_consensus.cc:348] T 0d923caa3f034a899050fe62dfc0c3d8 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } }
I20240502 21:26:58.144239 4674 raft_consensus.cc:729] T 0d923caa3f034a899050fe62dfc0c3d8 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: LEARNER
I20240502 21:26:58.144492 4674 consensus_queue.cc:260] T 0d923caa3f034a899050fe62dfc0c3d8 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } }
I20240502 21:26:58.145028 4674 ts_tablet_manager.cc:1398] T 0d923caa3f034a899050fe62dfc0c3d8 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240502 21:26:58.146340 2251 tablet_copy_service.cc:342] P d1406e92aec84ac585304e57bd37a40d: Request end of tablet copy session 08d7a0680d244b55b93d17f9ab57b84c-0d923caa3f034a899050fe62dfc0c3d8 received from {username='slave'} at 127.24.226.131:53089
I20240502 21:26:58.146468 2251 tablet_copy_service.cc:433] P d1406e92aec84ac585304e57bd37a40d: ending tablet copy session 08d7a0680d244b55b93d17f9ab57b84c-0d923caa3f034a899050fe62dfc0c3d8 on tablet 0d923caa3f034a899050fe62dfc0c3d8 with peer 08d7a0680d244b55b93d17f9ab57b84c
I20240502 21:26:58.170301 31391 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_extra_table_30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:58.183238 31905 tablet_service.cc:1467] Processing CreateTablet for tablet 7cd290642c7043439d6ab2da964dde78 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=17c1e25d551b47fe8c4b84ba12f7508c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:58.183465 2076 tablet_service.cc:1467] Processing CreateTablet for tablet 7cd290642c7043439d6ab2da964dde78 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=17c1e25d551b47fe8c4b84ba12f7508c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:58.183619 31905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cd290642c7043439d6ab2da964dde78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:58.183898 2076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cd290642c7043439d6ab2da964dde78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:58.188743 31775 tablet_service.cc:1467] Processing CreateTablet for tablet 7cd290642c7043439d6ab2da964dde78 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=17c1e25d551b47fe8c4b84ba12f7508c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:58.188753 1946 tablet_service.cc:1467] Processing CreateTablet for tablet c977ca41b72d4d45ac7146fbca350824 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=17c1e25d551b47fe8c4b84ba12f7508c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:58.189088 31775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cd290642c7043439d6ab2da964dde78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:58.189112 1946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c977ca41b72d4d45ac7146fbca350824. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:58.193380 1814 tablet_service.cc:1467] Processing CreateTablet for tablet 02c12b1f39a2415bbc137f983fb860b1 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=17c1e25d551b47fe8c4b84ba12f7508c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:58.193571 2077 tablet_service.cc:1467] Processing CreateTablet for tablet 02c12b1f39a2415bbc137f983fb860b1 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=17c1e25d551b47fe8c4b84ba12f7508c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:58.193823 1814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02c12b1f39a2415bbc137f983fb860b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:58.193905 2077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02c12b1f39a2415bbc137f983fb860b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:58.196576 1813 tablet_service.cc:1467] Processing CreateTablet for tablet c977ca41b72d4d45ac7146fbca350824 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=17c1e25d551b47fe8c4b84ba12f7508c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:58.196939 1813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c977ca41b72d4d45ac7146fbca350824. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:58.200809 2210 tablet_service.cc:1467] Processing CreateTablet for tablet 02c12b1f39a2415bbc137f983fb860b1 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=17c1e25d551b47fe8c4b84ba12f7508c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:58.201217 2210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02c12b1f39a2415bbc137f983fb860b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:58.205551 31906 tablet_service.cc:1467] Processing CreateTablet for tablet c977ca41b72d4d45ac7146fbca350824 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=17c1e25d551b47fe8c4b84ba12f7508c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:58.205976 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c977ca41b72d4d45ac7146fbca350824. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:58.206521 4706 tablet_bootstrap.cc:492] T 7cd290642c7043439d6ab2da964dde78 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:58.208263 4706 tablet_bootstrap.cc:654] T 7cd290642c7043439d6ab2da964dde78 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:58.208776 4705 tablet_bootstrap.cc:492] T c977ca41b72d4d45ac7146fbca350824 P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:58.211215 4705 tablet_bootstrap.cc:654] T c977ca41b72d4d45ac7146fbca350824 P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:58.217789 31391 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:26:58.219334 4243 raft_consensus.cc:979] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d: : Instructing follower 9dec645b473f40578e726f3e50e219bf to start an election
I20240502 21:26:58.219493 4243 raft_consensus.cc:1067] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Signalling peer 9dec645b473f40578e726f3e50e219bf to start an election
I20240502 21:26:58.222724 4708 tablet_bootstrap.cc:492] T 7cd290642c7043439d6ab2da964dde78 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:58.225766 4708 tablet_bootstrap.cc:654] T 7cd290642c7043439d6ab2da964dde78 P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:58.226214 4705 tablet_bootstrap.cc:492] T c977ca41b72d4d45ac7146fbca350824 P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:26:58.226382 4705 ts_tablet_manager.cc:1367] T c977ca41b72d4d45ac7146fbca350824 P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.006s
I20240502 21:26:58.227178 4705 raft_consensus.cc:348] T c977ca41b72d4d45ac7146fbca350824 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:58.227488 4705 raft_consensus.cc:374] T c977ca41b72d4d45ac7146fbca350824 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:58.227623 4705 raft_consensus.cc:729] T c977ca41b72d4d45ac7146fbca350824 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:26:58.227913 4705 consensus_queue.cc:260] T c977ca41b72d4d45ac7146fbca350824 P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:58.228628 4705 ts_tablet_manager.cc:1398] T c977ca41b72d4d45ac7146fbca350824 P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:26:58.232496 1965 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f13c3b9c4b7c471fa34156b781473ec4"
dest_uuid: "9dec645b473f40578e726f3e50e219bf"
from {username='slave'} at 127.24.226.136:40811
I20240502 21:26:58.240226 4709 ts_tablet_manager.cc:902] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 2d50f33557944bb480ea32199d9ee4f1: Initiating tablet copy from peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:58.241631 4708 tablet_bootstrap.cc:492] T 7cd290642c7043439d6ab2da964dde78 P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:58.242692 4706 tablet_bootstrap.cc:492] T 7cd290642c7043439d6ab2da964dde78 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:58.242799 4708 ts_tablet_manager.cc:1367] T 7cd290642c7043439d6ab2da964dde78 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.005s
I20240502 21:26:58.243877 4708 raft_consensus.cc:348] T 7cd290642c7043439d6ab2da964dde78 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:58.244290 4706 ts_tablet_manager.cc:1367] T 7cd290642c7043439d6ab2da964dde78 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.000s
I20240502 21:26:58.245800 4708 raft_consensus.cc:374] T 7cd290642c7043439d6ab2da964dde78 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:58.246157 4708 raft_consensus.cc:729] T 7cd290642c7043439d6ab2da964dde78 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:58.246717 4706 raft_consensus.cc:348] T 7cd290642c7043439d6ab2da964dde78 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:58.246810 4708 consensus_queue.cc:260] T 7cd290642c7043439d6ab2da964dde78 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:58.246937 4706 raft_consensus.cc:374] T 7cd290642c7043439d6ab2da964dde78 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:58.247022 4706 raft_consensus.cc:729] T 7cd290642c7043439d6ab2da964dde78 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:58.247309 4706 consensus_queue.cc:260] T 7cd290642c7043439d6ab2da964dde78 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:58.247874 4708 ts_tablet_manager.cc:1398] T 7cd290642c7043439d6ab2da964dde78 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240502 21:26:58.248032 4706 ts_tablet_manager.cc:1398] T 7cd290642c7043439d6ab2da964dde78 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:58.249841 4708 tablet_bootstrap.cc:492] T 02c12b1f39a2415bbc137f983fb860b1 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:58.251891 4708 tablet_bootstrap.cc:654] T 02c12b1f39a2415bbc137f983fb860b1 P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:58.256613 4706 tablet_bootstrap.cc:492] T c977ca41b72d4d45ac7146fbca350824 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:58.257813 1965 raft_consensus.cc:482] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:26:58.257946 1965 raft_consensus.cc:3037] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:58.258405 4706 tablet_bootstrap.cc:654] T c977ca41b72d4d45ac7146fbca350824 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:58.262753 4659 tablet.cc:1722] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:58.265621 1965 raft_consensus.cc:504] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } }
I20240502 21:26:58.268019 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13c3b9c4b7c471fa34156b781473ec4" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:26:58.268252 31536 raft_consensus.cc:3037] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:58.268467 4706 tablet_bootstrap.cc:492] T c977ca41b72d4d45ac7146fbca350824 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:26:58.268574 4706 ts_tablet_manager.cc:1367] T c977ca41b72d4d45ac7146fbca350824 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.004s
I20240502 21:26:58.268798 4660 tablet.cc:1722] T 929492106d2a4fdaad13a82b8ff36f7b P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:58.269254 4708 tablet_bootstrap.cc:492] T 02c12b1f39a2415bbc137f983fb860b1 P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:58.269280 4706 raft_consensus.cc:348] T c977ca41b72d4d45ac7146fbca350824 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:58.269412 2226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13c3b9c4b7c471fa34156b781473ec4" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:26:58.269516 4706 raft_consensus.cc:374] T c977ca41b72d4d45ac7146fbca350824 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:58.269572 4706 raft_consensus.cc:729] T c977ca41b72d4d45ac7146fbca350824 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:26:58.269616 2226 raft_consensus.cc:3032] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:26:58.269694 2226 raft_consensus.cc:729] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Running, Role: LEADER
I20240502 21:26:58.269754 4706 consensus_queue.cc:260] T c977ca41b72d4d45ac7146fbca350824 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:58.269992 2226 consensus_queue.cc:260] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } }
I20240502 21:26:58.270159 4708 ts_tablet_manager.cc:1367] T 02c12b1f39a2415bbc137f983fb860b1 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.004s
I20240502 21:26:58.270293 4706 ts_tablet_manager.cc:1398] T c977ca41b72d4d45ac7146fbca350824 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:58.270416 2226 raft_consensus.cc:3037] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:58.270802 4708 raft_consensus.cc:348] T 02c12b1f39a2415bbc137f983fb860b1 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:58.270974 4708 raft_consensus.cc:374] T 02c12b1f39a2415bbc137f983fb860b1 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:58.271042 4708 raft_consensus.cc:729] T 02c12b1f39a2415bbc137f983fb860b1 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:58.271186 31536 raft_consensus.cc:2454] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 2.
I20240502 21:26:58.272209 1965 leader_election.cc:288] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 2 election: Requested vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:26:58.273406 2226 raft_consensus.cc:2454] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 2.
I20240502 21:26:58.273640 4708 consensus_queue.cc:260] T 02c12b1f39a2415bbc137f983fb860b1 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:58.276895 4708 ts_tablet_manager.cc:1398] T 02c12b1f39a2415bbc137f983fb860b1 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240502 21:26:58.279145 1832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13c3b9c4b7c471fa34156b781473ec4" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:26:58.279402 1832 raft_consensus.cc:3037] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:26:58.281854 1832 raft_consensus.cc:2454] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 2.
I20240502 21:26:58.284967 4664 tablet.cc:1722] T 929492106d2a4fdaad13a82b8ff36f7b P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:58.286866 1901 leader_election.cc:302] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, 9dec645b473f40578e726f3e50e219bf, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:26:58.290539 3818 raft_consensus.cc:2781] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:26:58.290694 3818 raft_consensus.cc:686] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf [term 2 LEADER]: Becoming Leader. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Running, Role: LEADER
I20240502 21:26:58.290977 3818 consensus_queue.cc:237] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } }
I20240502 21:26:58.297013 31391 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table_new_name [id=cb776fab0f3e40559b48fb13dbded7d9] alter complete (version 3)
I20240502 21:26:58.305609 31387 catalog_manager.cc:5484] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf reported cstate change: term changed from 1 to 2, leader changed from d1406e92aec84ac585304e57bd37a40d (127.24.226.136) to 9dec645b473f40578e726f3e50e219bf (127.24.226.134). New cstate: current_term: 2 leader_uuid: "9dec645b473f40578e726f3e50e219bf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:58.322643 4709 tablet_copy_client.cc:308] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 2d50f33557944bb480ea32199d9ee4f1: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.131:40807
I20240502 21:26:58.323786 31817 tablet_copy_service.cc:140] P 08d7a0680d244b55b93d17f9ab57b84c: Received BeginTabletCopySession request for tablet 534b4d96df9b4bc5a9480b6c45eb11a5 from peer 2d50f33557944bb480ea32199d9ee4f1 ({username='slave'} at 127.24.226.132:44555)
I20240502 21:26:58.324009 31817 tablet_copy_service.cc:161] P 08d7a0680d244b55b93d17f9ab57b84c: Beginning new tablet copy session on tablet 534b4d96df9b4bc5a9480b6c45eb11a5 from peer 2d50f33557944bb480ea32199d9ee4f1 at {username='slave'} at 127.24.226.132:44555: session id = 2d50f33557944bb480ea32199d9ee4f1-534b4d96df9b4bc5a9480b6c45eb11a5
I20240502 21:26:58.341986 31817 tablet_copy_source_session.cc:205] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:58.344825 4709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 534b4d96df9b4bc5a9480b6c45eb11a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:58.356956 31666 raft_consensus.cc:1203] T 2c737c49a32b4f3ea5ef735fa254d48e P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:26:58.362766 4709 tablet_copy_client.cc:783] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 2d50f33557944bb480ea32199d9ee4f1: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:58.363108 4709 tablet_copy_client.cc:647] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 2d50f33557944bb480ea32199d9ee4f1: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:58.364903 4715 tablet_bootstrap.cc:492] T c977ca41b72d4d45ac7146fbca350824 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:58.367082 4715 tablet_bootstrap.cc:654] T c977ca41b72d4d45ac7146fbca350824 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:58.368253 4725 tablet_bootstrap.cc:492] T 02c12b1f39a2415bbc137f983fb860b1 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:58.371660 4709 tablet_copy_client.cc:523] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 2d50f33557944bb480ea32199d9ee4f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:58.375808 4725 tablet_bootstrap.cc:654] T 02c12b1f39a2415bbc137f983fb860b1 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:58.376327 4707 tablet_bootstrap.cc:492] T 7cd290642c7043439d6ab2da964dde78 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:58.377784 4709 tablet_bootstrap.cc:492] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:26:58.378113 4707 tablet_bootstrap.cc:654] T 7cd290642c7043439d6ab2da964dde78 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:58.380402 4718 ts_tablet_manager.cc:902] T 53b7eec870c44999b48f0bf744b2ffcd P 1e04157b98dd4fad98858c290b73c3db: Initiating tablet copy from peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:58.385011 4707 tablet_bootstrap.cc:492] T 7cd290642c7043439d6ab2da964dde78 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:26:58.387208 4725 tablet_bootstrap.cc:492] T 02c12b1f39a2415bbc137f983fb860b1 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:58.387784 4707 ts_tablet_manager.cc:1367] T 7cd290642c7043439d6ab2da964dde78 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240502 21:26:58.388583 4739 ts_tablet_manager.cc:902] T e0f96b56fb68439a97cf2aae4aa89b57 P 1ade0354c44c4c37ae480c7a94b11e1b: Initiating tablet copy from peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:58.388540 4707 raft_consensus.cc:348] T 7cd290642c7043439d6ab2da964dde78 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:58.388963 4718 tablet_copy_client.cc:308] T 53b7eec870c44999b48f0bf744b2ffcd P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.131:40807
I20240502 21:26:58.390095 4709 tablet_bootstrap.cc:492] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:58.390471 4709 tablet_bootstrap.cc:492] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap complete.
I20240502 21:26:58.390730 4709 ts_tablet_manager.cc:1367] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.001s
I20240502 21:26:58.392119 4725 ts_tablet_manager.cc:1367] T 02c12b1f39a2415bbc137f983fb860b1 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20240502 21:26:58.392994 4709 raft_consensus.cc:348] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } }
I20240502 21:26:58.393298 4709 raft_consensus.cc:729] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: LEARNER
I20240502 21:26:58.393503 4709 consensus_queue.cc:260] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } }
I20240502 21:26:58.391568 31817 tablet_copy_service.cc:140] P 08d7a0680d244b55b93d17f9ab57b84c: Received BeginTabletCopySession request for tablet 53b7eec870c44999b48f0bf744b2ffcd from peer 1e04157b98dd4fad98858c290b73c3db ({username='slave'} at 127.24.226.133:41593)
I20240502 21:26:58.393918 31817 tablet_copy_service.cc:161] P 08d7a0680d244b55b93d17f9ab57b84c: Beginning new tablet copy session on tablet 53b7eec870c44999b48f0bf744b2ffcd from peer 1e04157b98dd4fad98858c290b73c3db at {username='slave'} at 127.24.226.133:41593: session id = 1e04157b98dd4fad98858c290b73c3db-53b7eec870c44999b48f0bf744b2ffcd
I20240502 21:26:58.394305 4709 ts_tablet_manager.cc:1398] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240502 21:26:58.395148 31816 tablet_copy_service.cc:342] P 08d7a0680d244b55b93d17f9ab57b84c: Request end of tablet copy session 2d50f33557944bb480ea32199d9ee4f1-534b4d96df9b4bc5a9480b6c45eb11a5 received from {username='slave'} at 127.24.226.132:44555
I20240502 21:26:58.395304 31816 tablet_copy_service.cc:433] P 08d7a0680d244b55b93d17f9ab57b84c: ending tablet copy session 2d50f33557944bb480ea32199d9ee4f1-534b4d96df9b4bc5a9480b6c45eb11a5 on tablet 534b4d96df9b4bc5a9480b6c45eb11a5 with peer 2d50f33557944bb480ea32199d9ee4f1
I20240502 21:26:58.398335 31817 tablet_copy_source_session.cc:205] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:58.400249 4718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b7eec870c44999b48f0bf744b2ffcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:58.400887 4715 tablet_bootstrap.cc:492] T c977ca41b72d4d45ac7146fbca350824 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:58.401065 4715 ts_tablet_manager.cc:1367] T c977ca41b72d4d45ac7146fbca350824 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.000s
I20240502 21:26:58.402166 4715 raft_consensus.cc:348] T c977ca41b72d4d45ac7146fbca350824 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:58.402422 4715 raft_consensus.cc:374] T c977ca41b72d4d45ac7146fbca350824 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:58.402530 4715 raft_consensus.cc:729] T c977ca41b72d4d45ac7146fbca350824 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:58.402916 4715 consensus_queue.cc:260] T c977ca41b72d4d45ac7146fbca350824 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:58.405630 4715 ts_tablet_manager.cc:1398] T c977ca41b72d4d45ac7146fbca350824 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 21:26:58.406101 4715 tablet_bootstrap.cc:492] T 02c12b1f39a2415bbc137f983fb860b1 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:58.409152 4715 tablet_bootstrap.cc:654] T 02c12b1f39a2415bbc137f983fb860b1 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:58.422663 4707 raft_consensus.cc:374] T 7cd290642c7043439d6ab2da964dde78 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:58.422886 4707 raft_consensus.cc:729] T 7cd290642c7043439d6ab2da964dde78 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:58.423383 4707 consensus_queue.cc:260] T 7cd290642c7043439d6ab2da964dde78 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:58.424264 4707 ts_tablet_manager.cc:1398] T 7cd290642c7043439d6ab2da964dde78 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.036s user 0.002s sys 0.000s
I20240502 21:26:58.425851 4715 tablet_bootstrap.cc:492] T 02c12b1f39a2415bbc137f983fb860b1 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:58.425750 4725 raft_consensus.cc:348] T 02c12b1f39a2415bbc137f983fb860b1 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:58.426138 4715 ts_tablet_manager.cc:1367] T 02c12b1f39a2415bbc137f983fb860b1 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.006s
I20240502 21:26:58.426142 4725 raft_consensus.cc:374] T 02c12b1f39a2415bbc137f983fb860b1 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:58.426355 4725 raft_consensus.cc:729] T 02c12b1f39a2415bbc137f983fb860b1 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:58.426986 4725 consensus_queue.cc:260] T 02c12b1f39a2415bbc137f983fb860b1 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:58.427273 4715 raft_consensus.cc:348] T 02c12b1f39a2415bbc137f983fb860b1 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:58.427537 4715 raft_consensus.cc:374] T 02c12b1f39a2415bbc137f983fb860b1 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:58.427628 4715 raft_consensus.cc:729] T 02c12b1f39a2415bbc137f983fb860b1 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:58.429627 4715 consensus_queue.cc:260] T 02c12b1f39a2415bbc137f983fb860b1 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:58.434615 4715 ts_tablet_manager.cc:1398] T 02c12b1f39a2415bbc137f983fb860b1 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240502 21:26:58.436117 4725 ts_tablet_manager.cc:1398] T 02c12b1f39a2415bbc137f983fb860b1 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.044s user 0.002s sys 0.000s
I20240502 21:26:58.441339 4336 raft_consensus.cc:482] T 02c12b1f39a2415bbc137f983fb860b1 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:58.441502 4336 raft_consensus.cc:504] T 02c12b1f39a2415bbc137f983fb860b1 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:58.442471 4336 leader_election.cc:288] T 02c12b1f39a2415bbc137f983fb860b1 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:58.443740 2094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c12b1f39a2415bbc137f983fb860b1" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:58.444018 2094 raft_consensus.cc:2454] T 02c12b1f39a2415bbc137f983fb860b1 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 0.
I20240502 21:26:58.444875 2230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c12b1f39a2415bbc137f983fb860b1" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:26:58.445130 2230 raft_consensus.cc:2454] T 02c12b1f39a2415bbc137f983fb860b1 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 0.
I20240502 21:26:58.446115 1769 leader_election.cc:302] T 02c12b1f39a2415bbc137f983fb860b1 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:58.446532 31796 raft_consensus.cc:1203] T 0d923caa3f034a899050fe62dfc0c3d8 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:26:58.446995 4336 raft_consensus.cc:2781] T 02c12b1f39a2415bbc137f983fb860b1 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:58.447145 4336 raft_consensus.cc:482] T 02c12b1f39a2415bbc137f983fb860b1 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:58.447247 4336 raft_consensus.cc:3037] T 02c12b1f39a2415bbc137f983fb860b1 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:58.450590 4336 raft_consensus.cc:504] T 02c12b1f39a2415bbc137f983fb860b1 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:58.452667 2226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c12b1f39a2415bbc137f983fb860b1" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:26:58.452939 2226 raft_consensus.cc:3037] T 02c12b1f39a2415bbc137f983fb860b1 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:58.463786 2226 raft_consensus.cc:2454] T 02c12b1f39a2415bbc137f983fb860b1 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:26:58.464465 1769 leader_election.cc:302] T 02c12b1f39a2415bbc137f983fb860b1 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:26:58.465911 2094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c12b1f39a2415bbc137f983fb860b1" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:26:58.466022 4526 raft_consensus.cc:2781] T 02c12b1f39a2415bbc137f983fb860b1 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:58.466156 2094 raft_consensus.cc:3037] T 02c12b1f39a2415bbc137f983fb860b1 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:58.466203 4526 raft_consensus.cc:686] T 02c12b1f39a2415bbc137f983fb860b1 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Becoming Leader. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Running, Role: LEADER
I20240502 21:26:58.466499 4526 consensus_queue.cc:237] T 02c12b1f39a2415bbc137f983fb860b1 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:58.467576 4723 raft_consensus.cc:482] T 7cd290642c7043439d6ab2da964dde78 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:58.467739 4723 raft_consensus.cc:504] T 7cd290642c7043439d6ab2da964dde78 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:58.468466 4723 leader_election.cc:288] T 7cd290642c7043439d6ab2da964dde78 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:58.469578 31796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd290642c7043439d6ab2da964dde78" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:58.469801 31796 raft_consensus.cc:2454] T 7cd290642c7043439d6ab2da964dde78 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:58.470214 31862 leader_election.cc:302] T 7cd290642c7043439d6ab2da964dde78 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:58.470307 2095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd290642c7043439d6ab2da964dde78" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:58.470477 4723 raft_consensus.cc:2781] T 7cd290642c7043439d6ab2da964dde78 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:58.470551 4723 raft_consensus.cc:482] T 7cd290642c7043439d6ab2da964dde78 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:58.470538 2095 raft_consensus.cc:2454] T 7cd290642c7043439d6ab2da964dde78 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:26:58.470626 4723 raft_consensus.cc:3037] T 7cd290642c7043439d6ab2da964dde78 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:58.473109 4723 raft_consensus.cc:504] T 7cd290642c7043439d6ab2da964dde78 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:58.473754 4723 leader_election.cc:288] T 7cd290642c7043439d6ab2da964dde78 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Requested vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:58.474716 31796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd290642c7043439d6ab2da964dde78" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:58.475006 31796 raft_consensus.cc:3037] T 7cd290642c7043439d6ab2da964dde78 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:58.479378 31796 raft_consensus.cc:2454] T 7cd290642c7043439d6ab2da964dde78 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:26:58.480820 2095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd290642c7043439d6ab2da964dde78" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:26:58.481060 2095 raft_consensus.cc:3037] T 7cd290642c7043439d6ab2da964dde78 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:58.485221 31862 leader_election.cc:302] T 7cd290642c7043439d6ab2da964dde78 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:26:58.485477 4723 raft_consensus.cc:2781] T 7cd290642c7043439d6ab2da964dde78 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:58.485605 4723 raft_consensus.cc:686] T 7cd290642c7043439d6ab2da964dde78 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Running, Role: LEADER
I20240502 21:26:58.485886 4723 consensus_queue.cc:237] T 7cd290642c7043439d6ab2da964dde78 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:58.490306 4718 tablet_copy_client.cc:783] T 53b7eec870c44999b48f0bf744b2ffcd P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:58.494007 4718 tablet_copy_client.cc:647] T 53b7eec870c44999b48f0bf744b2ffcd P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:58.495676 2094 raft_consensus.cc:2454] T 02c12b1f39a2415bbc137f983fb860b1 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:26:58.499656 2095 raft_consensus.cc:2454] T 7cd290642c7043439d6ab2da964dde78 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:26:58.501550 4336 leader_election.cc:288] T 02c12b1f39a2415bbc137f983fb860b1 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Requested vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:58.510813 4739 tablet_copy_client.cc:308] T e0f96b56fb68439a97cf2aae4aa89b57 P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.131:40807
I20240502 21:26:58.511772 31817 tablet_copy_service.cc:140] P 08d7a0680d244b55b93d17f9ab57b84c: Received BeginTabletCopySession request for tablet e0f96b56fb68439a97cf2aae4aa89b57 from peer 1ade0354c44c4c37ae480c7a94b11e1b ({username='slave'} at 127.24.226.135:48019)
I20240502 21:26:58.511924 31817 tablet_copy_service.cc:161] P 08d7a0680d244b55b93d17f9ab57b84c: Beginning new tablet copy session on tablet e0f96b56fb68439a97cf2aae4aa89b57 from peer 1ade0354c44c4c37ae480c7a94b11e1b at {username='slave'} at 127.24.226.135:48019: session id = 1ade0354c44c4c37ae480c7a94b11e1b-e0f96b56fb68439a97cf2aae4aa89b57
I20240502 21:26:58.514132 31817 tablet_copy_source_session.cc:205] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:58.515676 4739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f96b56fb68439a97cf2aae4aa89b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:58.516578 4718 tablet_copy_client.cc:523] T 53b7eec870c44999b48f0bf744b2ffcd P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:58.524806 4739 tablet_copy_client.cc:783] T e0f96b56fb68439a97cf2aae4aa89b57 P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:58.525066 4739 tablet_copy_client.cc:647] T e0f96b56fb68439a97cf2aae4aa89b57 P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:58.528800 4739 tablet_copy_client.cc:523] T e0f96b56fb68439a97cf2aae4aa89b57 P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:58.531796 4739 tablet_bootstrap.cc:492] T e0f96b56fb68439a97cf2aae4aa89b57 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:58.533787 31387 catalog_manager.cc:5484] T 7cd290642c7043439d6ab2da964dde78 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132). New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:58.540478 31391 catalog_manager.cc:5484] T 02c12b1f39a2415bbc137f983fb860b1 P 1e04157b98dd4fad98858c290b73c3db reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e04157b98dd4fad98858c290b73c3db (127.24.226.133). New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:58.548421 3818 raft_consensus.cc:482] T c977ca41b72d4d45ac7146fbca350824 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:58.548601 3818 raft_consensus.cc:504] T c977ca41b72d4d45ac7146fbca350824 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:58.549000 4718 tablet_bootstrap.cc:492] T 53b7eec870c44999b48f0bf744b2ffcd P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:58.549305 3818 leader_election.cc:288] T c977ca41b72d4d45ac7146fbca350824 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:58.549703 31923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c977ca41b72d4d45ac7146fbca350824" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:26:58.549927 31923 raft_consensus.cc:2454] T c977ca41b72d4d45ac7146fbca350824 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 0.
I20240502 21:26:58.550493 1902 leader_election.cc:302] T c977ca41b72d4d45ac7146fbca350824 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:26:58.551079 3818 raft_consensus.cc:2781] T c977ca41b72d4d45ac7146fbca350824 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:58.551192 3818 raft_consensus.cc:482] T c977ca41b72d4d45ac7146fbca350824 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:58.551290 3818 raft_consensus.cc:3037] T c977ca41b72d4d45ac7146fbca350824 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:58.549641 1832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c977ca41b72d4d45ac7146fbca350824" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:26:58.552266 1832 raft_consensus.cc:2454] T c977ca41b72d4d45ac7146fbca350824 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 0.
I20240502 21:26:58.553750 3818 raft_consensus.cc:504] T c977ca41b72d4d45ac7146fbca350824 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:58.554390 3818 leader_election.cc:288] T c977ca41b72d4d45ac7146fbca350824 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 election: Requested vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:26:58.554785 1832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c977ca41b72d4d45ac7146fbca350824" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:26:58.555006 1832 raft_consensus.cc:3037] T c977ca41b72d4d45ac7146fbca350824 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:58.557745 31923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c977ca41b72d4d45ac7146fbca350824" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:26:58.557914 31923 raft_consensus.cc:3037] T c977ca41b72d4d45ac7146fbca350824 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:58.558245 1832 raft_consensus.cc:2454] T c977ca41b72d4d45ac7146fbca350824 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 1.
I20240502 21:26:58.558785 1901 leader_election.cc:302] T c977ca41b72d4d45ac7146fbca350824 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:26:58.559154 3818 raft_consensus.cc:2781] T c977ca41b72d4d45ac7146fbca350824 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:58.559286 3818 raft_consensus.cc:686] T c977ca41b72d4d45ac7146fbca350824 P 9dec645b473f40578e726f3e50e219bf [term 1 LEADER]: Becoming Leader. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Running, Role: LEADER
I20240502 21:26:58.559526 3818 consensus_queue.cc:237] T c977ca41b72d4d45ac7146fbca350824 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:26:58.560257 31923 raft_consensus.cc:2454] T c977ca41b72d4d45ac7146fbca350824 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 1.
I20240502 21:26:58.564165 31387 catalog_manager.cc:5484] T c977ca41b72d4d45ac7146fbca350824 P 9dec645b473f40578e726f3e50e219bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dec645b473f40578e726f3e50e219bf (127.24.226.134). New cstate: current_term: 1 leader_uuid: "9dec645b473f40578e726f3e50e219bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:58.584378 4739 tablet_bootstrap.cc:492] T e0f96b56fb68439a97cf2aae4aa89b57 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:58.584832 4739 tablet_bootstrap.cc:492] T e0f96b56fb68439a97cf2aae4aa89b57 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap complete.
I20240502 21:26:58.585166 4739 ts_tablet_manager.cc:1367] T e0f96b56fb68439a97cf2aae4aa89b57 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.053s user 0.041s sys 0.011s
I20240502 21:26:58.586249 4739 raft_consensus.cc:348] T e0f96b56fb68439a97cf2aae4aa89b57 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } }
I20240502 21:26:58.586498 4739 raft_consensus.cc:729] T e0f96b56fb68439a97cf2aae4aa89b57 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: LEARNER
I20240502 21:26:58.586669 4739 consensus_queue.cc:260] T e0f96b56fb68439a97cf2aae4aa89b57 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } }
I20240502 21:26:58.588210 4739 ts_tablet_manager.cc:1398] T e0f96b56fb68439a97cf2aae4aa89b57 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:26:58.588742 31817 tablet_copy_service.cc:342] P 08d7a0680d244b55b93d17f9ab57b84c: Request end of tablet copy session 1ade0354c44c4c37ae480c7a94b11e1b-e0f96b56fb68439a97cf2aae4aa89b57 received from {username='slave'} at 127.24.226.135:48019
I20240502 21:26:58.588835 31817 tablet_copy_service.cc:433] P 08d7a0680d244b55b93d17f9ab57b84c: ending tablet copy session 1ade0354c44c4c37ae480c7a94b11e1b-e0f96b56fb68439a97cf2aae4aa89b57 on tablet e0f96b56fb68439a97cf2aae4aa89b57 with peer 1ade0354c44c4c37ae480c7a94b11e1b
I20240502 21:26:58.621196 31387 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:58.621419 31387 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:58.623899 31533 raft_consensus.cc:1203] T 2068521c26dc44f9ba60e274eb039b22 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:26:58.632761 31387 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 929492106d2a4fdaad13a82b8ff36f7b
I20240502 21:26:58.634860 2076 tablet_service.cc:1514] Processing DeleteTablet for tablet 929492106d2a4fdaad13a82b8ff36f7b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:58 UTC) from {username='slave'} at 127.0.0.1:36740
I20240502 21:26:58.634900 1946 tablet_service.cc:1514] Processing DeleteTablet for tablet 929492106d2a4fdaad13a82b8ff36f7b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:58 UTC) from {username='slave'} at 127.0.0.1:33002
I20240502 21:26:58.635828 2210 tablet_service.cc:1514] Processing DeleteTablet for tablet 929492106d2a4fdaad13a82b8ff36f7b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:58 UTC) from {username='slave'} at 127.0.0.1:36984
I20240502 21:26:58.651170 4761 tablet_replica.cc:330] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf: stopping tablet replica
I20240502 21:26:58.651584 4761 raft_consensus.cc:2229] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:58.656273 4762 tablet_replica.cc:330] T 929492106d2a4fdaad13a82b8ff36f7b P d1406e92aec84ac585304e57bd37a40d: stopping tablet replica
I20240502 21:26:58.656805 4761 raft_consensus.cc:2258] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:58.656984 4762 raft_consensus.cc:2229] T 929492106d2a4fdaad13a82b8ff36f7b P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:58.657310 4762 raft_consensus.cc:2258] T 929492106d2a4fdaad13a82b8ff36f7b P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:58.658157 4761 ts_tablet_manager.cc:1875] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:58.658422 4762 ts_tablet_manager.cc:1875] T 929492106d2a4fdaad13a82b8ff36f7b P d1406e92aec84ac585304e57bd37a40d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:58.660801 31923 raft_consensus.cc:1203] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:26:58.666230 4760 tablet_replica.cc:330] T 929492106d2a4fdaad13a82b8ff36f7b P 1ade0354c44c4c37ae480c7a94b11e1b: stopping tablet replica
I20240502 21:26:58.666464 4760 raft_consensus.cc:2229] T 929492106d2a4fdaad13a82b8ff36f7b P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:58.666682 4760 raft_consensus.cc:2258] T 929492106d2a4fdaad13a82b8ff36f7b P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:58.669417 4760 ts_tablet_manager.cc:1875] T 929492106d2a4fdaad13a82b8ff36f7b P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:58.676967 4760 ts_tablet_manager.cc:1888] T 929492106d2a4fdaad13a82b8ff36f7b P 1ade0354c44c4c37ae480c7a94b11e1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:58.677165 4760 log.cc:1196] T 929492106d2a4fdaad13a82b8ff36f7b P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal/wals/929492106d2a4fdaad13a82b8ff36f7b
I20240502 21:26:58.677791 4760 ts_tablet_manager.cc:1909] T 929492106d2a4fdaad13a82b8ff36f7b P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting consensus metadata
I20240502 21:26:58.677846 4762 ts_tablet_manager.cc:1888] T 929492106d2a4fdaad13a82b8ff36f7b P d1406e92aec84ac585304e57bd37a40d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:58.678047 4762 log.cc:1196] T 929492106d2a4fdaad13a82b8ff36f7b P d1406e92aec84ac585304e57bd37a40d: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal/wals/929492106d2a4fdaad13a82b8ff36f7b
I20240502 21:26:58.678728 4762 ts_tablet_manager.cc:1909] T 929492106d2a4fdaad13a82b8ff36f7b P d1406e92aec84ac585304e57bd37a40d: Deleting consensus metadata
I20240502 21:26:58.679199 31381 catalog_manager.cc:4830] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): tablet 929492106d2a4fdaad13a82b8ff36f7b (table rebalancer_dynamic_table_new_name [id=cb776fab0f3e40559b48fb13dbded7d9]) successfully deleted
I20240502 21:26:58.679734 4761 ts_tablet_manager.cc:1888] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:58.679876 4761 log.cc:1196] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal/wals/929492106d2a4fdaad13a82b8ff36f7b
I20240502 21:26:58.680369 4761 ts_tablet_manager.cc:1909] T 929492106d2a4fdaad13a82b8ff36f7b P 9dec645b473f40578e726f3e50e219bf: Deleting consensus metadata
I20240502 21:26:58.681360 31379 catalog_manager.cc:4830] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): tablet 929492106d2a4fdaad13a82b8ff36f7b (table rebalancer_dynamic_table_new_name [id=cb776fab0f3e40559b48fb13dbded7d9]) successfully deleted
I20240502 21:26:58.682541 31381 catalog_manager.cc:4830] TS d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): tablet 929492106d2a4fdaad13a82b8ff36f7b (table rebalancer_dynamic_table_new_name [id=cb776fab0f3e40559b48fb13dbded7d9]) successfully deleted
I20240502 21:26:58.690263 4718 tablet_bootstrap.cc:492] T 53b7eec870c44999b48f0bf744b2ffcd P 1e04157b98dd4fad98858c290b73c3db: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:58.690675 4718 tablet_bootstrap.cc:492] T 53b7eec870c44999b48f0bf744b2ffcd P 1e04157b98dd4fad98858c290b73c3db: Bootstrap complete.
I20240502 21:26:58.690950 4718 ts_tablet_manager.cc:1367] T 53b7eec870c44999b48f0bf744b2ffcd P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.142s user 0.058s sys 0.000s
I20240502 21:26:58.691622 4718 raft_consensus.cc:348] T 53b7eec870c44999b48f0bf744b2ffcd P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } }
I20240502 21:26:58.691900 4718 raft_consensus.cc:729] T 53b7eec870c44999b48f0bf744b2ffcd P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: LEARNER
I20240502 21:26:58.692126 4718 consensus_queue.cc:260] T 53b7eec870c44999b48f0bf744b2ffcd P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } }
I20240502 21:26:58.692664 4718 ts_tablet_manager.cc:1398] T 53b7eec870c44999b48f0bf744b2ffcd P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 21:26:58.693359 31817 tablet_copy_service.cc:342] P 08d7a0680d244b55b93d17f9ab57b84c: Request end of tablet copy session 1e04157b98dd4fad98858c290b73c3db-53b7eec870c44999b48f0bf744b2ffcd received from {username='slave'} at 127.24.226.133:41593
I20240502 21:26:58.693495 31817 tablet_copy_service.cc:433] P 08d7a0680d244b55b93d17f9ab57b84c: ending tablet copy session 1e04157b98dd4fad98858c290b73c3db-53b7eec870c44999b48f0bf744b2ffcd on tablet 53b7eec870c44999b48f0bf744b2ffcd with peer 1e04157b98dd4fad98858c290b73c3db
I20240502 21:26:58.707320 1832 raft_consensus.cc:1261] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db [term 2 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:26:58.707839 3818 consensus_queue.cc:1035] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:58.727620 2226 consensus_queue.cc:237] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } }
I20240502 21:26:58.730324 1832 raft_consensus.cc:1203] T 53b7eec870c44999b48f0bf744b2ffcd P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:26:58.732566 1964 raft_consensus.cc:1261] T c5ba5610b754466fb1e12db451295809 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:58.732793 31923 raft_consensus.cc:1261] T c5ba5610b754466fb1e12db451295809 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:26:58.733003 2163 consensus_peers.cc:485] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d -> Peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): Couldn't send request to peer 08d7a0680d244b55b93d17f9ab57b84c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5ba5610b754466fb1e12db451295809. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:58.733304 4498 consensus_queue.cc:1035] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:58.735143 4498 consensus_queue.cc:1035] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:58.735070 31387 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:26:58.737776 4498 raft_consensus.cc:2932] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } } }
I20240502 21:26:58.741398 31923 raft_consensus.cc:2932] T c5ba5610b754466fb1e12db451295809 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } } }
I20240502 21:26:58.743827 31386 catalog_manager.cc:5484] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d reported cstate change: config changed from index -1 to 2, NON_VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) added. New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:58.745210 31647 tablet_service.cc:1467] Processing CreateTablet for tablet eb5970b1ec3b4c50b3a59d600fc0696f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=075b835e0b4942c08a6620c055c49882]), partition=
I20240502 21:26:58.745635 31647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb5970b1ec3b4c50b3a59d600fc0696f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:58.745788 2076 tablet_service.cc:1467] Processing CreateTablet for tablet eb5970b1ec3b4c50b3a59d600fc0696f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=075b835e0b4942c08a6620c055c49882]), partition=
I20240502 21:26:58.746135 2076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb5970b1ec3b4c50b3a59d600fc0696f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:58.746264 1946 tablet_service.cc:1467] Processing CreateTablet for tablet eb5970b1ec3b4c50b3a59d600fc0696f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=075b835e0b4942c08a6620c055c49882]), partition=
I20240502 21:26:58.746649 1946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb5970b1ec3b4c50b3a59d600fc0696f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:58.754154 4708 tablet_bootstrap.cc:492] T eb5970b1ec3b4c50b3a59d600fc0696f P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:58.755663 4708 tablet_bootstrap.cc:654] T eb5970b1ec3b4c50b3a59d600fc0696f P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:58.755720 4705 tablet_bootstrap.cc:492] T eb5970b1ec3b4c50b3a59d600fc0696f P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:58.757653 4705 tablet_bootstrap.cc:654] T eb5970b1ec3b4c50b3a59d600fc0696f P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:58.767976 4705 tablet_bootstrap.cc:492] T eb5970b1ec3b4c50b3a59d600fc0696f P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:26:58.768232 4705 ts_tablet_manager.cc:1367] T eb5970b1ec3b4c50b3a59d600fc0696f P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.004s
I20240502 21:26:58.769213 4705 raft_consensus.cc:348] T eb5970b1ec3b4c50b3a59d600fc0696f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:58.769627 4705 raft_consensus.cc:374] T eb5970b1ec3b4c50b3a59d600fc0696f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:58.769775 4705 raft_consensus.cc:729] T eb5970b1ec3b4c50b3a59d600fc0696f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:26:58.770203 4708 tablet_bootstrap.cc:492] T eb5970b1ec3b4c50b3a59d600fc0696f P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:58.770157 4705 consensus_queue.cc:260] T eb5970b1ec3b4c50b3a59d600fc0696f P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:58.770401 4708 ts_tablet_manager.cc:1367] T eb5970b1ec3b4c50b3a59d600fc0696f P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.001s
I20240502 21:26:58.771035 4708 raft_consensus.cc:348] T eb5970b1ec3b4c50b3a59d600fc0696f P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:58.771230 4708 raft_consensus.cc:374] T eb5970b1ec3b4c50b3a59d600fc0696f P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:58.771314 4708 raft_consensus.cc:729] T eb5970b1ec3b4c50b3a59d600fc0696f P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:58.771528 4708 consensus_queue.cc:260] T eb5970b1ec3b4c50b3a59d600fc0696f P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:58.772197 4705 ts_tablet_manager.cc:1398] T eb5970b1ec3b4c50b3a59d600fc0696f P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:26:58.775836 4708 ts_tablet_manager.cc:1398] T eb5970b1ec3b4c50b3a59d600fc0696f P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 21:26:58.784662 2094 raft_consensus.cc:1203] T e0f96b56fb68439a97cf2aae4aa89b57 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:26:58.785569 4783 tablet_bootstrap.cc:492] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:58.789773 4783 tablet_bootstrap.cc:654] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:58.791450 1964 raft_consensus.cc:2932] T c5ba5610b754466fb1e12db451295809 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } } }
I20240502 21:26:58.805711 4783 tablet_bootstrap.cc:492] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:58.805879 4783 ts_tablet_manager.cc:1367] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20240502 21:26:58.806510 4783 raft_consensus.cc:348] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:58.806740 4783 raft_consensus.cc:374] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:58.806838 4783 raft_consensus.cc:729] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:58.807104 4783 consensus_queue.cc:260] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:58.812701 4783 ts_tablet_manager.cc:1398] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 21:26:58.816538 3859 raft_consensus.cc:482] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:58.816649 3859 raft_consensus.cc:504] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:58.817299 3859 leader_election.cc:288] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:58.818071 1964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5970b1ec3b4c50b3a59d600fc0696f" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:26:58.818286 1964 raft_consensus.cc:2454] T eb5970b1ec3b4c50b3a59d600fc0696f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:26:58.818315 2094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5970b1ec3b4c50b3a59d600fc0696f" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:58.818514 2094 raft_consensus.cc:2454] T eb5970b1ec3b4c50b3a59d600fc0696f P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:26:58.818706 31600 leader_election.cc:302] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:26:58.818934 3859 raft_consensus.cc:2781] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:58.818992 3859 raft_consensus.cc:482] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:58.819042 3859 raft_consensus.cc:3037] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:58.821506 3859 raft_consensus.cc:504] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:58.822440 2094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5970b1ec3b4c50b3a59d600fc0696f" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:26:58.822619 2094 raft_consensus.cc:3037] T eb5970b1ec3b4c50b3a59d600fc0696f P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:58.824514 3859 leader_election.cc:288] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Requested vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:26:58.825428 2094 raft_consensus.cc:2454] T eb5970b1ec3b4c50b3a59d600fc0696f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:58.825819 31602 leader_election.cc:302] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 1ade0354c44c4c37ae480c7a94b11e1b; no voters:
I20240502 21:26:58.826134 3859 raft_consensus.cc:2781] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:58.826506 3859 raft_consensus.cc:686] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Becoming Leader. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Running, Role: LEADER
I20240502 21:26:58.826753 3859 consensus_queue.cc:237] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:26:58.833794 2230 raft_consensus.cc:1261] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [term 2 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:26:58.834511 1964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb5970b1ec3b4c50b3a59d600fc0696f" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf"
I20240502 21:26:58.834486 4581 consensus_queue.cc:1035] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:58.834733 1964 raft_consensus.cc:3037] T eb5970b1ec3b4c50b3a59d600fc0696f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:58.838166 1964 raft_consensus.cc:2454] T eb5970b1ec3b4c50b3a59d600fc0696f P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:26:58.839043 4744 raft_consensus.cc:1050] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db: attempting to promote NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada to VOTER
I20240502 21:26:58.839877 4744 consensus_queue.cc:237] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } }
I20240502 21:26:58.845899 2094 raft_consensus.cc:1261] T 2c737c49a32b4f3ea5ef735fa254d48e P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:58.846009 31795 raft_consensus.cc:1261] T 2c737c49a32b4f3ea5ef735fa254d48e P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:58.846755 4336 consensus_queue.cc:1035] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:58.847065 31666 raft_consensus.cc:1261] T 2c737c49a32b4f3ea5ef735fa254d48e P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:58.847206 4336 consensus_queue.cc:1035] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:58.847759 4336 consensus_queue.cc:1035] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:58.856168 31665 consensus_queue.cc:237] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } }
I20240502 21:26:58.859550 4526 raft_consensus.cc:2932] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:26:58.864857 31667 raft_consensus.cc:2932] T 2c737c49a32b4f3ea5ef735fa254d48e P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:26:58.867522 31536 raft_consensus.cc:1261] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f [term 2 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:26:58.870024 3818 consensus_queue.cc:1035] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:58.880945 31795 raft_consensus.cc:2932] T 2c737c49a32b4f3ea5ef735fa254d48e P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:26:58.881619 4679 consensus_queue.cc:1035] T 02c12b1f39a2415bbc137f983fb860b1 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:58.883425 31391 catalog_manager.cc:5484] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db reported cstate change: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:58.898281 1964 consensus_queue.cc:237] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } }
I20240502 21:26:58.903481 31536 raft_consensus.cc:1261] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f [term 2 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 21:26:58.904167 4581 consensus_queue.cc:1035] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:58.906375 3818 raft_consensus.cc:2932] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:58.908010 4407 raft_consensus.cc:1050] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d: attempting to promote NON_VOTER 08d7a0680d244b55b93d17f9ab57b84c to VOTER
I20240502 21:26:58.908872 4407 consensus_queue.cc:237] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } }
I20240502 21:26:58.910547 31379 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f13c3b9c4b7c471fa34156b781473ec4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:58.912142 31536 raft_consensus.cc:2932] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:58.916484 2095 raft_consensus.cc:2932] T 2c737c49a32b4f3ea5ef735fa254d48e P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:26:58.923638 31795 raft_consensus.cc:1261] T 0d923caa3f034a899050fe62dfc0c3d8 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEARNER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:58.924466 4499 consensus_queue.cc:1035] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:58.927764 4744 consensus_queue.cc:1035] T 02c12b1f39a2415bbc137f983fb860b1 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:26:58.930086 31602 consensus_peers.cc:485] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): Couldn't send request to peer 1ade0354c44c4c37ae480c7a94b11e1b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 94ba99a5798f4d5c9442590066896f93. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:58.931133 31797 raft_consensus.cc:1261] T 94ba99a5798f4d5c9442590066896f93 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:58.931622 4432 consensus_queue.cc:1035] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:58.931718 31925 raft_consensus.cc:1261] T 94ba99a5798f4d5c9442590066896f93 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:58.932248 4432 consensus_queue.cc:1035] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:58.938433 31392 catalog_manager.cc:5484] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130). New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:58.944353 31392 catalog_manager.cc:5484] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf reported cstate change: config changed from index 3 to 5, VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) evicted. New cstate: current_term: 2 leader_uuid: "9dec645b473f40578e726f3e50e219bf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:58.946132 1834 raft_consensus.cc:1261] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db [term 2 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 21:26:58.948680 1964 raft_consensus.cc:1261] T 0d923caa3f034a899050fe62dfc0c3d8 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:58.947547 4801 consensus_queue.cc:1035] T f13c3b9c4b7c471fa34156b781473ec4 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:58.949333 4499 consensus_queue.cc:1035] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:58.951112 2094 raft_consensus.cc:1261] T 0d923caa3f034a899050fe62dfc0c3d8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:58.950911 1832 raft_consensus.cc:2932] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:58.951686 4499 consensus_queue.cc:1035] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:58.953922 2210 tablet_service.cc:1514] Processing DeleteTablet for tablet f13c3b9c4b7c471fa34156b781473ec4 with delete_type TABLET_DATA_TOMBSTONED (TS d1406e92aec84ac585304e57bd37a40d not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:36984
I20240502 21:26:58.954550 4762 tablet_replica.cc:330] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d: stopping tablet replica
I20240502 21:26:58.954855 4762 raft_consensus.cc:2229] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:58.955067 4762 raft_consensus.cc:2258] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:58.956300 4762 ts_tablet_manager.cc:1875] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:58.960223 4810 raft_consensus.cc:2932] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } } }
I20240502 21:26:58.960592 31796 raft_consensus.cc:2932] T 94ba99a5798f4d5c9442590066896f93 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } } }
I20240502 21:26:58.962560 31925 raft_consensus.cc:2932] T 94ba99a5798f4d5c9442590066896f93 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } } }
I20240502 21:26:58.970692 31389 catalog_manager.cc:5484] T 94ba99a5798f4d5c9442590066896f93 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } } }
I20240502 21:26:58.977270 4499 raft_consensus.cc:2932] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:26:58.978461 1964 raft_consensus.cc:2932] T 0d923caa3f034a899050fe62dfc0c3d8 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:26:58.981997 31796 raft_consensus.cc:2932] T 0d923caa3f034a899050fe62dfc0c3d8 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:26:58.982600 31386 catalog_manager.cc:5484] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d reported cstate change: config changed from index 2 to 3, 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:58.983690 4762 ts_tablet_manager.cc:1888] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 21:26:58.983846 4762 log.cc:1196] T f13c3b9c4b7c471fa34156b781473ec4 P d1406e92aec84ac585304e57bd37a40d: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal/wals/f13c3b9c4b7c471fa34156b781473ec4
I20240502 21:26:58.987296 2224 consensus_queue.cc:237] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } }
I20240502 21:26:58.989318 2095 raft_consensus.cc:1261] T 0d923caa3f034a899050fe62dfc0c3d8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:58.990152 4499 consensus_queue.cc:1035] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:58.991354 31381 catalog_manager.cc:4830] TS d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): tablet f13c3b9c4b7c471fa34156b781473ec4 (table rebalancer_extra_table_12 [id=daf389fa76d84011ad57fe7156cfa6a8]) successfully deleted
I20240502 21:26:58.991194 2095 raft_consensus.cc:2932] T 0d923caa3f034a899050fe62dfc0c3d8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:26:58.995445 31797 raft_consensus.cc:1261] T 0d923caa3f034a899050fe62dfc0c3d8 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:58.996182 4499 consensus_queue.cc:1035] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:58.997673 4498 raft_consensus.cc:2932] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:26:58.999109 2095 raft_consensus.cc:2932] T 0d923caa3f034a899050fe62dfc0c3d8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:26:59.003939 31381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0d923caa3f034a899050fe62dfc0c3d8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:59.005537 31392 catalog_manager.cc:5484] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d reported cstate change: config changed from index 3 to 4, VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) evicted. New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:59.009325 1946 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d923caa3f034a899050fe62dfc0c3d8 with delete_type TABLET_DATA_TOMBSTONED (TS 9dec645b473f40578e726f3e50e219bf not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33002
I20240502 21:26:59.009680 4761 tablet_replica.cc:330] T 0d923caa3f034a899050fe62dfc0c3d8 P 9dec645b473f40578e726f3e50e219bf: stopping tablet replica
I20240502 21:26:59.009757 4723 consensus_queue.cc:1035] T 7cd290642c7043439d6ab2da964dde78 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:59.009954 4761 raft_consensus.cc:2229] T 0d923caa3f034a899050fe62dfc0c3d8 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:59.010180 4761 raft_consensus.cc:2258] T 0d923caa3f034a899050fe62dfc0c3d8 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:59.009891 31797 raft_consensus.cc:2932] T 0d923caa3f034a899050fe62dfc0c3d8 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:26:59.011297 4761 ts_tablet_manager.cc:1875] T 0d923caa3f034a899050fe62dfc0c3d8 P 9dec645b473f40578e726f3e50e219bf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:59.017614 4761 ts_tablet_manager.cc:1888] T 0d923caa3f034a899050fe62dfc0c3d8 P 9dec645b473f40578e726f3e50e219bf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:26:59.018529 4761 log.cc:1196] T 0d923caa3f034a899050fe62dfc0c3d8 P 9dec645b473f40578e726f3e50e219bf: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal/wals/0d923caa3f034a899050fe62dfc0c3d8
I20240502 21:26:59.019786 31379 catalog_manager.cc:4830] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): tablet 0d923caa3f034a899050fe62dfc0c3d8 (table rebalancer_extra_table_14 [id=d8ac6ebe732c415bb64d1aecfeadc621]) successfully deleted
I20240502 21:26:59.036239 1964 consensus_queue.cc:237] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } }
I20240502 21:26:59.047111 31536 raft_consensus.cc:1261] T b20c2bb60f6149458a62ee443b8cf3ab P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:59.048005 2224 raft_consensus.cc:1261] T b20c2bb60f6149458a62ee443b8cf3ab P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:26:59.049855 1902 consensus_peers.cc:485] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf -> Peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): Couldn't send request to peer 05f4fac41c4f4959b9b316cb112d0ada. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b20c2bb60f6149458a62ee443b8cf3ab. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:59.050067 4581 consensus_queue.cc:1035] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:59.054441 4801 consensus_queue.cc:1035] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:59.055868 4407 raft_consensus.cc:1050] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d: attempting to promote NON_VOTER a44f41b0b6644fb6aa29476aed8c964f to VOTER
I20240502 21:26:59.056046 4801 consensus_queue.cc:1035] T c977ca41b72d4d45ac7146fbca350824 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:59.056689 4407 consensus_queue.cc:237] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } }
I20240502 21:26:59.067963 31391 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:26:59.074244 31392 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_extra_table_32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:26:59.085806 2076 tablet_service.cc:1467] Processing CreateTablet for tablet 939c34669f6447f9a2a88da769d0f84f (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=f630d42dfa594ae0bddc6c021ec7af14]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:59.086305 2076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 939c34669f6447f9a2a88da769d0f84f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.093853 1946 tablet_service.cc:1467] Processing CreateTablet for tablet 939c34669f6447f9a2a88da769d0f84f (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=f630d42dfa594ae0bddc6c021ec7af14]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:59.093884 1813 tablet_service.cc:1467] Processing CreateTablet for tablet 939c34669f6447f9a2a88da769d0f84f (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=f630d42dfa594ae0bddc6c021ec7af14]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:59.094311 1946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 939c34669f6447f9a2a88da769d0f84f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.094360 1813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 939c34669f6447f9a2a88da769d0f84f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.100379 1814 tablet_service.cc:1467] Processing CreateTablet for tablet a8684b1ca3a642688a232afaf1b28f96 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=f630d42dfa594ae0bddc6c021ec7af14]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:59.100824 1814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8684b1ca3a642688a232afaf1b28f96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.102478 31533 raft_consensus.cc:1261] T 2068521c26dc44f9ba60e274eb039b22 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEARNER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:59.103137 4498 consensus_queue.cc:1035] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:59.104605 2210 tablet_service.cc:1467] Processing CreateTablet for tablet a8684b1ca3a642688a232afaf1b28f96 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=f630d42dfa594ae0bddc6c021ec7af14]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:59.105036 2210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8684b1ca3a642688a232afaf1b28f96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.107101 1945 tablet_service.cc:1467] Processing CreateTablet for tablet a8684b1ca3a642688a232afaf1b28f96 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=f630d42dfa594ae0bddc6c021ec7af14]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:59.107194 4723 consensus_queue.cc:1035] T 7cd290642c7043439d6ab2da964dde78 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:59.107446 1945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8684b1ca3a642688a232afaf1b28f96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.112095 4581 consensus_queue.cc:1035] T c977ca41b72d4d45ac7146fbca350824 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:59.115396 4705 tablet_bootstrap.cc:492] T a8684b1ca3a642688a232afaf1b28f96 P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:59.117038 4705 tablet_bootstrap.cc:654] T a8684b1ca3a642688a232afaf1b28f96 P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.120191 1812 tablet_service.cc:1467] Processing CreateTablet for tablet e9140579721c4b3fbba776e4e16e5d9d (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=f630d42dfa594ae0bddc6c021ec7af14]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:59.120643 1812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9140579721c4b3fbba776e4e16e5d9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.126350 2077 tablet_service.cc:1467] Processing CreateTablet for tablet e9140579721c4b3fbba776e4e16e5d9d (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=f630d42dfa594ae0bddc6c021ec7af14]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:59.126739 2077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9140579721c4b3fbba776e4e16e5d9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.128865 2208 tablet_service.cc:1467] Processing CreateTablet for tablet e9140579721c4b3fbba776e4e16e5d9d (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=f630d42dfa594ae0bddc6c021ec7af14]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:26:59.129310 2208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9140579721c4b3fbba776e4e16e5d9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.136050 4708 tablet_bootstrap.cc:492] T 939c34669f6447f9a2a88da769d0f84f P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:59.137918 4708 tablet_bootstrap.cc:654] T 939c34669f6447f9a2a88da769d0f84f P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.144213 4705 tablet_bootstrap.cc:492] T a8684b1ca3a642688a232afaf1b28f96 P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:26:59.144447 4705 ts_tablet_manager.cc:1367] T a8684b1ca3a642688a232afaf1b28f96 P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.004s
I20240502 21:26:59.145676 4705 raft_consensus.cc:348] T a8684b1ca3a642688a232afaf1b28f96 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:59.146301 4705 raft_consensus.cc:374] T a8684b1ca3a642688a232afaf1b28f96 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.146497 4705 raft_consensus.cc:729] T a8684b1ca3a642688a232afaf1b28f96 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.146783 4705 consensus_queue.cc:260] T a8684b1ca3a642688a232afaf1b28f96 P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:59.147395 4705 ts_tablet_manager.cc:1398] T a8684b1ca3a642688a232afaf1b28f96 P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:26:59.147004 4581 raft_consensus.cc:2932] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } } }
I20240502 21:26:59.147815 4705 tablet_bootstrap.cc:492] T 939c34669f6447f9a2a88da769d0f84f P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:26:59.151651 2094 raft_consensus.cc:1261] T eb5970b1ec3b4c50b3a59d600fc0696f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:59.152240 4432 consensus_queue.cc:1035] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:59.152139 2226 raft_consensus.cc:2932] T b20c2bb60f6149458a62ee443b8cf3ab P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } } }
I20240502 21:26:59.156967 4559 raft_consensus.cc:1050] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c: attempting to promote NON_VOTER 1e04157b98dd4fad98858c290b73c3db to VOTER
I20240502 21:26:59.158000 4559 consensus_queue.cc:237] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } }
I20240502 21:26:59.158581 31536 raft_consensus.cc:2932] T b20c2bb60f6149458a62ee443b8cf3ab P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } } }
I20240502 21:26:59.163012 31392 catalog_manager.cc:5484] T b20c2bb60f6149458a62ee443b8cf3ab P d1406e92aec84ac585304e57bd37a40d reported cstate change: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New cstate: current_term: 1 leader_uuid: "9dec645b473f40578e726f3e50e219bf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } } }
I20240502 21:26:59.168190 4840 ts_tablet_manager.cc:902] T c5ba5610b754466fb1e12db451295809 P 08d7a0680d244b55b93d17f9ab57b84c: Initiating tablet copy from peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:59.176774 4841 tablet.cc:1722] T eb5970b1ec3b4c50b3a59d600fc0696f P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:59.177232 1964 raft_consensus.cc:1261] T eb5970b1ec3b4c50b3a59d600fc0696f P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:59.178515 31535 raft_consensus.cc:1261] T 53b7eec870c44999b48f0bf744b2ffcd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:26:59.179061 3946 consensus_queue.cc:1035] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:26:59.180300 4841 tablet.cc:1620] T eb5970b1ec3b4c50b3a59d600fc0696f P 1ade0354c44c4c37ae480c7a94b11e1b: MemRowSet was empty: no flush needed.
I20240502 21:26:59.181437 2226 raft_consensus.cc:1261] T 53b7eec870c44999b48f0bf744b2ffcd P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:26:59.182174 4167 consensus_queue.cc:1035] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:26:59.185235 1832 raft_consensus.cc:1261] T 53b7eec870c44999b48f0bf744b2ffcd P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:26:59.185855 4432 consensus_queue.cc:1035] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:59.186532 4167 consensus_queue.cc:1035] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:26:59.189195 3946 raft_consensus.cc:2932] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:59.193204 2226 raft_consensus.cc:2932] T 53b7eec870c44999b48f0bf744b2ffcd P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:59.210917 4705 tablet_bootstrap.cc:654] T 939c34669f6447f9a2a88da769d0f84f P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.211930 31392 catalog_manager.cc:5484] T 53b7eec870c44999b48f0bf744b2ffcd P d1406e92aec84ac585304e57bd37a40d reported cstate change: config changed from index 7 to 8, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:59.237238 1965 raft_consensus.cc:1261] T 2068521c26dc44f9ba60e274eb039b22 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:59.237905 4498 consensus_queue.cc:1035] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:59.240424 31386 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=075b835e0b4942c08a6620c055c49882] alter complete (version 1)
I20240502 21:26:59.241739 4705 tablet_bootstrap.cc:492] T 939c34669f6447f9a2a88da769d0f84f P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:26:59.242123 2095 raft_consensus.cc:1261] T 2068521c26dc44f9ba60e274eb039b22 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:59.243450 4407 consensus_queue.cc:1035] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:59.244210 4705 ts_tablet_manager.cc:1367] T 939c34669f6447f9a2a88da769d0f84f P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.097s user 0.003s sys 0.002s
I20240502 21:26:59.245100 4498 raft_consensus.cc:2932] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:59.245687 4705 raft_consensus.cc:348] T 939c34669f6447f9a2a88da769d0f84f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:59.245887 4705 raft_consensus.cc:374] T 939c34669f6447f9a2a88da769d0f84f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.245951 4705 raft_consensus.cc:729] T 939c34669f6447f9a2a88da769d0f84f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.249154 1965 raft_consensus.cc:2932] T 2068521c26dc44f9ba60e274eb039b22 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:59.249569 4705 consensus_queue.cc:260] T 939c34669f6447f9a2a88da769d0f84f P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:59.251682 4705 ts_tablet_manager.cc:1398] T 939c34669f6447f9a2a88da769d0f84f P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240502 21:26:59.261637 31392 catalog_manager.cc:5484] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d reported cstate change: config changed from index 2 to 3, a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:59.263779 31533 raft_consensus.cc:2932] T 2068521c26dc44f9ba60e274eb039b22 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:59.278393 2226 consensus_queue.cc:237] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } }
I20240502 21:26:59.281919 1965 raft_consensus.cc:1261] T 2068521c26dc44f9ba60e274eb039b22 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:59.282999 4407 consensus_queue.cc:1035] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:59.285573 4499 raft_consensus.cc:2932] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:59.286165 1965 raft_consensus.cc:2932] T 2068521c26dc44f9ba60e274eb039b22 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:59.290472 31381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2068521c26dc44f9ba60e274eb039b22 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:59.292352 4839 tablet.cc:1722] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:59.293169 4839 tablet.cc:1620] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada: MemRowSet was empty: no flush needed.
I20240502 21:26:59.297662 4832 tablet_bootstrap.cc:492] T a8684b1ca3a642688a232afaf1b28f96 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:59.300163 4832 tablet_bootstrap.cc:654] T a8684b1ca3a642688a232afaf1b28f96 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.303398 4835 tablet_bootstrap.cc:492] T a8684b1ca3a642688a232afaf1b28f96 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:59.314419 31391 catalog_manager.cc:5484] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d reported cstate change: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:59.321497 1834 consensus_queue.cc:237] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:26:59.330888 2077 tablet_service.cc:1514] Processing DeleteTablet for tablet 2068521c26dc44f9ba60e274eb039b22 with delete_type TABLET_DATA_TOMBSTONED (TS 1ade0354c44c4c37ae480c7a94b11e1b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36740
W20240502 21:26:59.337985 1767 consensus_peers.cc:485] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e6657723fe84e749dea768de44638e7. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:59.339221 31535 raft_consensus.cc:1261] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:59.339529 2094 raft_consensus.cc:1261] T 3e6657723fe84e749dea768de44638e7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:59.340165 4744 consensus_queue.cc:1035] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:59.340586 4708 tablet_bootstrap.cc:492] T 939c34669f6447f9a2a88da769d0f84f P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:59.341204 4744 consensus_queue.cc:1035] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:59.343035 31392 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:26:59.351099 4744 raft_consensus.cc:2932] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:59.351992 4708 ts_tablet_manager.cc:1367] T 939c34669f6447f9a2a88da769d0f84f P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.205s user 0.003s sys 0.008s
I20240502 21:26:59.356261 31535 raft_consensus.cc:2932] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:59.357717 4835 tablet_bootstrap.cc:654] T a8684b1ca3a642688a232afaf1b28f96 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.357378 1832 raft_consensus.cc:2932] T 53b7eec870c44999b48f0bf744b2ffcd P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:59.361109 31536 raft_consensus.cc:1261] T 2068521c26dc44f9ba60e274eb039b22 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:59.363477 4407 consensus_queue.cc:1035] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:59.365039 31536 raft_consensus.cc:2932] T 2068521c26dc44f9ba60e274eb039b22 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:26:59.369308 4801 raft_consensus.cc:482] T 939c34669f6447f9a2a88da769d0f84f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:59.369474 4801 raft_consensus.cc:504] T 939c34669f6447f9a2a88da769d0f84f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:59.370963 4801 leader_election.cc:288] T 939c34669f6447f9a2a88da769d0f84f P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:26:59.371649 1834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "939c34669f6447f9a2a88da769d0f84f" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
W20240502 21:26:59.372316 1901 leader_election.cc:341] T 939c34669f6447f9a2a88da769d0f84f P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:59.373286 31393 catalog_manager.cc:5576] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645) does not have the latest schema for tablet eb5970b1ec3b4c50b3a59d600fc0696f (table rebalancer_dynamic_table [id=075b835e0b4942c08a6620c055c49882]). Expected version 2 got 1
I20240502 21:26:59.374215 4832 tablet_bootstrap.cc:492] T a8684b1ca3a642688a232afaf1b28f96 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:59.375422 4858 tablet.cc:1722] T eb5970b1ec3b4c50b3a59d600fc0696f P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:26:59.376144 4832 ts_tablet_manager.cc:1367] T a8684b1ca3a642688a232afaf1b28f96 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.079s user 0.004s sys 0.001s
I20240502 21:26:59.377104 4832 raft_consensus.cc:348] T a8684b1ca3a642688a232afaf1b28f96 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:59.380805 31391 catalog_manager.cc:5484] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: config changed from index -1 to 2, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:59.382419 31393 catalog_manager.cc:5576] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753) does not have the latest schema for tablet eb5970b1ec3b4c50b3a59d600fc0696f (table rebalancer_dynamic_table [id=075b835e0b4942c08a6620c055c49882]). Expected version 2 got 0
I20240502 21:26:59.388636 31393 catalog_manager.cc:5576] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645) does not have the latest schema for tablet eb5970b1ec3b4c50b3a59d600fc0696f (table rebalancer_dynamic_table [id=075b835e0b4942c08a6620c055c49882]). Expected version 2 got 1
I20240502 21:26:59.394776 31393 catalog_manager.cc:5576] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753) does not have the latest schema for tablet eb5970b1ec3b4c50b3a59d600fc0696f (table rebalancer_dynamic_table [id=075b835e0b4942c08a6620c055c49882]). Expected version 2 got 1
I20240502 21:26:59.399933 4835 tablet_bootstrap.cc:492] T a8684b1ca3a642688a232afaf1b28f96 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:59.400104 4835 ts_tablet_manager.cc:1367] T a8684b1ca3a642688a232afaf1b28f96 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.097s user 0.004s sys 0.000s
I20240502 21:26:59.401154 4835 raft_consensus.cc:348] T a8684b1ca3a642688a232afaf1b28f96 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:59.401399 4835 raft_consensus.cc:374] T a8684b1ca3a642688a232afaf1b28f96 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.401511 4835 raft_consensus.cc:729] T a8684b1ca3a642688a232afaf1b28f96 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.401767 4835 consensus_queue.cc:260] T a8684b1ca3a642688a232afaf1b28f96 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:59.401895 4868 tablet.cc:1722] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:59.402671 4868 tablet.cc:1620] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada: MemRowSet was empty: no flush needed.
I20240502 21:26:59.402751 4858 tablet.cc:1620] T eb5970b1ec3b4c50b3a59d600fc0696f P 9dec645b473f40578e726f3e50e219bf: MemRowSet was empty: no flush needed.
I20240502 21:26:59.405896 4832 raft_consensus.cc:374] T a8684b1ca3a642688a232afaf1b28f96 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.406356 4832 raft_consensus.cc:729] T a8684b1ca3a642688a232afaf1b28f96 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.417713 4835 ts_tablet_manager.cc:1398] T a8684b1ca3a642688a232afaf1b28f96 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240502 21:26:59.420508 31392 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=075b835e0b4942c08a6620c055c49882] alter complete (version 2)
I20240502 21:26:59.430373 31536 raft_consensus.cc:2932] T 53b7eec870c44999b48f0bf744b2ffcd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:59.443312 2093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "939c34669f6447f9a2a88da769d0f84f" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:59.444304 4841 tablet.cc:1722] T eb5970b1ec3b4c50b3a59d600fc0696f P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:59.444806 4841 tablet.cc:1620] T eb5970b1ec3b4c50b3a59d600fc0696f P 1ade0354c44c4c37ae480c7a94b11e1b: MemRowSet was empty: no flush needed.
I20240502 21:26:59.445223 3946 raft_consensus.cc:1050] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c: attempting to promote NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b to VOTER
I20240502 21:26:59.445997 3946 consensus_queue.cc:237] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } }
I20240502 21:26:59.449985 4858 tablet.cc:1722] T eb5970b1ec3b4c50b3a59d600fc0696f P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:26:59.450533 4858 tablet.cc:1620] T eb5970b1ec3b4c50b3a59d600fc0696f P 9dec645b473f40578e726f3e50e219bf: MemRowSet was empty: no flush needed.
I20240502 21:26:59.468144 2097 raft_consensus.cc:1261] T e0f96b56fb68439a97cf2aae4aa89b57 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:59.468685 3946 consensus_queue.cc:1035] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 21:26:59.472354 1901 leader_election.cc:341] T 939c34669f6447f9a2a88da769d0f84f P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:59.472537 1901 leader_election.cc:302] T 939c34669f6447f9a2a88da769d0f84f P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dec645b473f40578e726f3e50e219bf; no voters: 1ade0354c44c4c37ae480c7a94b11e1b, 1e04157b98dd4fad98858c290b73c3db
I20240502 21:26:59.472666 4801 raft_consensus.cc:482] T a8684b1ca3a642688a232afaf1b28f96 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:59.472790 4801 raft_consensus.cc:504] T a8684b1ca3a642688a232afaf1b28f96 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:59.472995 3818 raft_consensus.cc:2726] T 939c34669f6447f9a2a88da769d0f84f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:26:59.473536 4801 leader_election.cc:288] T a8684b1ca3a642688a232afaf1b28f96 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:59.474335 1832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8684b1ca3a642688a232afaf1b28f96" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:26:59.474720 4708 raft_consensus.cc:348] T 939c34669f6447f9a2a88da769d0f84f P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:59.474936 4708 raft_consensus.cc:374] T 939c34669f6447f9a2a88da769d0f84f P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.475284 4708 raft_consensus.cc:729] T 939c34669f6447f9a2a88da769d0f84f P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.475586 4708 consensus_queue.cc:260] T 939c34669f6447f9a2a88da769d0f84f P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:59.476438 4708 ts_tablet_manager.cc:1398] T 939c34669f6447f9a2a88da769d0f84f P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.124s user 0.001s sys 0.001s
I20240502 21:26:59.476787 4708 tablet_bootstrap.cc:492] T e9140579721c4b3fbba776e4e16e5d9d P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:59.482448 31796 consensus_queue.cc:237] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } }
I20240502 21:26:59.484633 1834 raft_consensus.cc:1261] T 53b7eec870c44999b48f0bf744b2ffcd P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:26:59.485954 4863 consensus_queue.cc:1035] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:26:59.490304 4863 raft_consensus.cc:2932] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:59.495090 31923 raft_consensus.cc:1261] T e0f96b56fb68439a97cf2aae4aa89b57 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:59.494875 1834 raft_consensus.cc:2932] T 53b7eec870c44999b48f0bf744b2ffcd P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:59.496193 31667 raft_consensus.cc:1261] T e0f96b56fb68439a97cf2aae4aa89b57 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:26:59.497663 4167 consensus_queue.cc:1035] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:59.497851 4867 consensus_queue.cc:1035] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:26:59.498041 4559 raft_consensus.cc:1050] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c: attempting to promote NON_VOTER 2d50f33557944bb480ea32199d9ee4f1 to VOTER
I20240502 21:26:59.498709 4559 consensus_queue.cc:237] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } }
I20240502 21:26:59.522082 31924 raft_consensus.cc:1261] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEARNER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:59.522596 4559 consensus_queue.cc:1035] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:59.529089 4832 consensus_queue.cc:260] T a8684b1ca3a642688a232afaf1b28f96 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:59.529462 1832 raft_consensus.cc:2454] T a8684b1ca3a642688a232afaf1b28f96 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 0.
I20240502 21:26:59.530041 4832 ts_tablet_manager.cc:1398] T a8684b1ca3a642688a232afaf1b28f96 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.154s user 0.000s sys 0.002s
I20240502 21:26:59.530215 1901 leader_election.cc:302] T a8684b1ca3a642688a232afaf1b28f96 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:26:59.530493 4832 tablet_bootstrap.cc:492] T 939c34669f6447f9a2a88da769d0f84f P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:59.531184 4801 raft_consensus.cc:2781] T a8684b1ca3a642688a232afaf1b28f96 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:59.531301 4801 raft_consensus.cc:482] T a8684b1ca3a642688a232afaf1b28f96 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:59.531409 4801 raft_consensus.cc:3037] T a8684b1ca3a642688a232afaf1b28f96 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.534963 4801 raft_consensus.cc:504] T a8684b1ca3a642688a232afaf1b28f96 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:59.536278 1832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8684b1ca3a642688a232afaf1b28f96" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:26:59.536504 1832 raft_consensus.cc:3037] T a8684b1ca3a642688a232afaf1b28f96 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.536922 4801 leader_election.cc:288] T a8684b1ca3a642688a232afaf1b28f96 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 election: Requested vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:59.435178 31391 catalog_manager.cc:5576] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753) does not have the latest schema for tablet eb5970b1ec3b4c50b3a59d600fc0696f (table rebalancer_dynamic_table [id=075b835e0b4942c08a6620c055c49882]). Expected version 2 got 1
I20240502 21:26:59.474318 2226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8684b1ca3a642688a232afaf1b28f96" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:26:59.508416 31381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 53b7eec870c44999b48f0bf744b2ffcd with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:59.511598 31386 catalog_manager.cc:5484] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index 8 to 9, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:59.539155 2226 raft_consensus.cc:2454] T a8684b1ca3a642688a232afaf1b28f96 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 0.
I20240502 21:26:59.485553 2224 raft_consensus.cc:1261] T 53b7eec870c44999b48f0bf744b2ffcd P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:26:59.522197 2230 raft_consensus.cc:1261] T 534b4d96df9b4bc5a9480b6c45eb11a5 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:59.522477 31387 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:26:59.540315 4559 consensus_queue.cc:1035] T 53b7eec870c44999b48f0bf744b2ffcd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:26:59.536759 2225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8684b1ca3a642688a232afaf1b28f96" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:26:59.540719 4559 consensus_queue.cc:1035] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:59.540773 2225 raft_consensus.cc:3037] T a8684b1ca3a642688a232afaf1b28f96 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.541929 2230 raft_consensus.cc:2932] T 53b7eec870c44999b48f0bf744b2ffcd P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:26:59.539196 1832 raft_consensus.cc:2454] T a8684b1ca3a642688a232afaf1b28f96 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 1.
I20240502 21:26:59.543632 2225 raft_consensus.cc:2454] T a8684b1ca3a642688a232afaf1b28f96 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 1.
I20240502 21:26:59.543926 1901 leader_election.cc:302] T a8684b1ca3a642688a232afaf1b28f96 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:26:59.546046 4801 raft_consensus.cc:2781] T a8684b1ca3a642688a232afaf1b28f96 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:59.546492 4801 raft_consensus.cc:686] T a8684b1ca3a642688a232afaf1b28f96 P 9dec645b473f40578e726f3e50e219bf [term 1 LEADER]: Becoming Leader. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Running, Role: LEADER
I20240502 21:26:59.546772 4801 consensus_queue.cc:237] T a8684b1ca3a642688a232afaf1b28f96 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:26:59.547431 31514 tablet_service.cc:1514] Processing DeleteTablet for tablet 53b7eec870c44999b48f0bf744b2ffcd with delete_type TABLET_DATA_TOMBSTONED (TS a44f41b0b6644fb6aa29476aed8c964f not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:45194
I20240502 21:26:59.549016 4835 tablet_bootstrap.cc:492] T e9140579721c4b3fbba776e4e16e5d9d P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:26:59.550588 4835 tablet_bootstrap.cc:654] T e9140579721c4b3fbba776e4e16e5d9d P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.555648 4868 tablet.cc:1722] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:59.556531 4863 raft_consensus.cc:2932] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:59.523849 2093 raft_consensus.cc:1261] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:26:59.528954 4861 tablet_replica.cc:330] T 2068521c26dc44f9ba60e274eb039b22 P 1ade0354c44c4c37ae480c7a94b11e1b: stopping tablet replica
I20240502 21:26:59.558023 4861 raft_consensus.cc:2229] T 2068521c26dc44f9ba60e274eb039b22 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:59.558225 4861 pending_rounds.cc:62] T 2068521c26dc44f9ba60e274eb039b22 P 1ade0354c44c4c37ae480c7a94b11e1b: Trying to abort 1 pending ops.
I20240502 21:26:59.558319 4861 pending_rounds.cc:68] T 2068521c26dc44f9ba60e274eb039b22 P 1ade0354c44c4c37ae480c7a94b11e1b: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7023350657254133760 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "2068521c26dc44f9ba60e274eb039b22" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } } }
I20240502 21:26:59.558848 4861 raft_consensus.cc:2894] T 2068521c26dc44f9ba60e274eb039b22 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240502 21:26:59.558974 4861 raft_consensus.cc:2866] T 2068521c26dc44f9ba60e274eb039b22 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 21:26:59.559092 4861 raft_consensus.cc:2258] T 2068521c26dc44f9ba60e274eb039b22 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:59.560750 4861 ts_tablet_manager.cc:1875] T 2068521c26dc44f9ba60e274eb039b22 P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:59.560995 4859 consensus_queue.cc:1035] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:26:59.571966 4861 ts_tablet_manager.cc:1888] T 2068521c26dc44f9ba60e274eb039b22 P 1ade0354c44c4c37ae480c7a94b11e1b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:26:59.572155 4861 log.cc:1196] T 2068521c26dc44f9ba60e274eb039b22 P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal/wals/2068521c26dc44f9ba60e274eb039b22
I20240502 21:26:59.574534 31391 catalog_manager.cc:5576] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753) does not have the latest schema for tablet eb5970b1ec3b4c50b3a59d600fc0696f (table rebalancer_dynamic_table_new_name [id=075b835e0b4942c08a6620c055c49882]). Expected version 3 got 2
I20240502 21:26:59.575109 31667 raft_consensus.cc:2932] T e0f96b56fb68439a97cf2aae4aa89b57 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:59.579725 2097 raft_consensus.cc:2932] T 3e6657723fe84e749dea768de44638e7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:26:59.578964 2096 raft_consensus.cc:2932] T e0f96b56fb68439a97cf2aae4aa89b57 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:59.581808 31388 catalog_manager.cc:5484] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:59.585268 31393 catalog_manager.cc:5484] T a8684b1ca3a642688a232afaf1b28f96 P 9dec645b473f40578e726f3e50e219bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dec645b473f40578e726f3e50e219bf (127.24.226.134). New cstate: current_term: 1 leader_uuid: "9dec645b473f40578e726f3e50e219bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: HEALTHY } } }
I20240502 21:26:59.586796 4874 ts_tablet_manager.cc:902] T 94ba99a5798f4d5c9442590066896f93 P 1ade0354c44c4c37ae480c7a94b11e1b: Initiating tablet copy from peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:26:59.588706 4883 tablet_replica.cc:330] T 53b7eec870c44999b48f0bf744b2ffcd P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:26:59.588936 4883 raft_consensus.cc:2229] T 53b7eec870c44999b48f0bf744b2ffcd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:59.589367 31923 raft_consensus.cc:2932] T e0f96b56fb68439a97cf2aae4aa89b57 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:59.591284 4883 raft_consensus.cc:2258] T 53b7eec870c44999b48f0bf744b2ffcd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:59.597784 31797 consensus_queue.cc:237] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } }
I20240502 21:26:59.599504 4883 ts_tablet_manager.cc:1875] T 53b7eec870c44999b48f0bf744b2ffcd P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:59.600092 31925 raft_consensus.cc:1261] T e0f96b56fb68439a97cf2aae4aa89b57 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:59.600144 2096 raft_consensus.cc:1261] T e0f96b56fb68439a97cf2aae4aa89b57 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:26:59.600630 4859 consensus_queue.cc:1035] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:59.601641 4841 tablet.cc:1722] T eb5970b1ec3b4c50b3a59d600fc0696f P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:59.603715 4863 consensus_queue.cc:1035] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:26:59.604238 4863 raft_consensus.cc:2932] T e0f96b56fb68439a97cf2aae4aa89b57 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:59.608466 2092 raft_consensus.cc:2932] T e0f96b56fb68439a97cf2aae4aa89b57 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:59.612540 31392 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table_new_name [id=075b835e0b4942c08a6620c055c49882] alter complete (version 3)
I20240502 21:26:59.612993 4840 tablet_copy_client.cc:308] T c5ba5610b754466fb1e12db451295809 P 08d7a0680d244b55b93d17f9ab57b84c: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.136:34907
I20240502 21:26:59.613696 2251 tablet_copy_service.cc:140] P d1406e92aec84ac585304e57bd37a40d: Received BeginTabletCopySession request for tablet c5ba5610b754466fb1e12db451295809 from peer 08d7a0680d244b55b93d17f9ab57b84c ({username='slave'} at 127.24.226.131:53089)
I20240502 21:26:59.613924 2251 tablet_copy_service.cc:161] P d1406e92aec84ac585304e57bd37a40d: Beginning new tablet copy session on tablet c5ba5610b754466fb1e12db451295809 from peer 08d7a0680d244b55b93d17f9ab57b84c at {username='slave'} at 127.24.226.131:53089: session id = 08d7a0680d244b55b93d17f9ab57b84c-c5ba5610b754466fb1e12db451295809
I20240502 21:26:59.618804 2251 tablet_copy_source_session.cc:205] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:59.620463 4840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5ba5610b754466fb1e12db451295809. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.620574 4859 raft_consensus.cc:2932] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:26:59.622762 31392 catalog_manager.cc:5484] T e0f96b56fb68439a97cf2aae4aa89b57 P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: config changed from index 5 to 6, VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) evicted. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:59.626737 31924 raft_consensus.cc:2932] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:26:59.627604 31647 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f96b56fb68439a97cf2aae4aa89b57 with delete_type TABLET_DATA_TOMBSTONED (TS 05f4fac41c4f4959b9b316cb112d0ada not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:59.634714 31381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e0f96b56fb68439a97cf2aae4aa89b57 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:59.635574 4840 tablet_copy_client.cc:783] T c5ba5610b754466fb1e12db451295809 P 08d7a0680d244b55b93d17f9ab57b84c: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:59.636065 4840 tablet_copy_client.cc:647] T c5ba5610b754466fb1e12db451295809 P 08d7a0680d244b55b93d17f9ab57b84c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:59.636927 4871 tablet.cc:1722] T eb5970b1ec3b4c50b3a59d600fc0696f P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:26:59.640799 31381 catalog_manager.cc:4830] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): tablet 2068521c26dc44f9ba60e274eb039b22 (table rebalancer_extra_table_20 [id=aae794f4f92e4a4b9f2610d66d67525d]) successfully deleted
I20240502 21:26:59.641949 4840 tablet_copy_client.cc:523] T c5ba5610b754466fb1e12db451295809 P 08d7a0680d244b55b93d17f9ab57b84c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:59.478453 4708 tablet_bootstrap.cc:654] T e9140579721c4b3fbba776e4e16e5d9d P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.647290 4840 tablet_bootstrap.cc:492] T c5ba5610b754466fb1e12db451295809 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:59.647193 2225 raft_consensus.cc:2932] T 534b4d96df9b4bc5a9480b6c45eb11a5 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:26:59.649292 4832 tablet_bootstrap.cc:654] T 939c34669f6447f9a2a88da769d0f84f P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.651316 31393 catalog_manager.cc:5484] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index 2 to 3, 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:59.649717 4835 tablet_bootstrap.cc:492] T e9140579721c4b3fbba776e4e16e5d9d P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:26:59.655558 4835 ts_tablet_manager.cc:1367] T e9140579721c4b3fbba776e4e16e5d9d P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.107s user 0.003s sys 0.000s
I20240502 21:26:59.656415 4835 raft_consensus.cc:348] T e9140579721c4b3fbba776e4e16e5d9d P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:59.654354 31923 raft_consensus.cc:2932] T e0f96b56fb68439a97cf2aae4aa89b57 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:26:59.656790 4835 raft_consensus.cc:374] T e9140579721c4b3fbba776e4e16e5d9d P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.656968 4835 raft_consensus.cc:729] T e9140579721c4b3fbba776e4e16e5d9d P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.657959 4835 consensus_queue.cc:260] T e9140579721c4b3fbba776e4e16e5d9d P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:59.659025 4840 tablet_bootstrap.cc:492] T c5ba5610b754466fb1e12db451295809 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:59.659408 4840 tablet_bootstrap.cc:492] T c5ba5610b754466fb1e12db451295809 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap complete.
I20240502 21:26:59.659655 4840 ts_tablet_manager.cc:1367] T c5ba5610b754466fb1e12db451295809 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.002s
I20240502 21:26:59.659845 4835 ts_tablet_manager.cc:1398] T e9140579721c4b3fbba776e4e16e5d9d P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 21:26:59.663539 4840 raft_consensus.cc:348] T c5ba5610b754466fb1e12db451295809 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } }
I20240502 21:26:59.664140 4840 raft_consensus.cc:729] T c5ba5610b754466fb1e12db451295809 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: LEARNER
I20240502 21:26:59.664407 4840 consensus_queue.cc:260] T c5ba5610b754466fb1e12db451295809 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } }
I20240502 21:26:59.667114 4840 ts_tablet_manager.cc:1398] T c5ba5610b754466fb1e12db451295809 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.007s user 0.000s sys 0.004s
I20240502 21:26:59.672690 2251 tablet_copy_service.cc:342] P d1406e92aec84ac585304e57bd37a40d: Request end of tablet copy session 08d7a0680d244b55b93d17f9ab57b84c-c5ba5610b754466fb1e12db451295809 received from {username='slave'} at 127.24.226.131:53089
I20240502 21:26:59.672838 2251 tablet_copy_service.cc:433] P d1406e92aec84ac585304e57bd37a40d: ending tablet copy session 08d7a0680d244b55b93d17f9ab57b84c-c5ba5610b754466fb1e12db451295809 on tablet c5ba5610b754466fb1e12db451295809 with peer 08d7a0680d244b55b93d17f9ab57b84c
I20240502 21:26:59.677677 2094 raft_consensus.cc:2932] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:26:59.679054 4892 tablet_replica.cc:330] T e0f96b56fb68439a97cf2aae4aa89b57 P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:59.679304 4892 raft_consensus.cc:2229] T e0f96b56fb68439a97cf2aae4aa89b57 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:59.679700 4892 raft_consensus.cc:2258] T e0f96b56fb68439a97cf2aae4aa89b57 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:59.681063 4892 ts_tablet_manager.cc:1875] T e0f96b56fb68439a97cf2aae4aa89b57 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:59.688201 31797 consensus_queue.cc:237] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } }
I20240502 21:26:59.691383 31925 raft_consensus.cc:1261] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:59.691900 4859 consensus_queue.cc:1035] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:59.694320 4863 raft_consensus.cc:2932] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:26:59.694654 2093 raft_consensus.cc:1261] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:26:59.694867 31925 raft_consensus.cc:2932] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:26:59.695351 4559 consensus_queue.cc:1035] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:26:59.696561 2093 raft_consensus.cc:2932] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:26:59.697360 4708 tablet_bootstrap.cc:492] T e9140579721c4b3fbba776e4e16e5d9d P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:26:59.697507 4708 ts_tablet_manager.cc:1367] T e9140579721c4b3fbba776e4e16e5d9d P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.221s user 0.003s sys 0.000s
I20240502 21:26:59.698479 31381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 534b4d96df9b4bc5a9480b6c45eb11a5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:59.698438 4708 raft_consensus.cc:348] T e9140579721c4b3fbba776e4e16e5d9d P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:59.699254 4708 raft_consensus.cc:374] T e9140579721c4b3fbba776e4e16e5d9d P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.699460 4708 raft_consensus.cc:729] T e9140579721c4b3fbba776e4e16e5d9d P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.701668 4708 consensus_queue.cc:260] T e9140579721c4b3fbba776e4e16e5d9d P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:59.702514 31797 raft_consensus.cc:1203] T c5ba5610b754466fb1e12db451295809 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:26:59.702641 4708 ts_tablet_manager.cc:1398] T e9140579721c4b3fbba776e4e16e5d9d P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:26:59.705255 4832 tablet_bootstrap.cc:492] T 939c34669f6447f9a2a88da769d0f84f P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:59.708855 31386 catalog_manager.cc:5484] T 534b4d96df9b4bc5a9480b6c45eb11a5 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: config changed from index 3 to 4, VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) evicted. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:26:59.715860 2210 tablet_service.cc:1514] Processing DeleteTablet for tablet 534b4d96df9b4bc5a9480b6c45eb11a5 with delete_type TABLET_DATA_TOMBSTONED (TS d1406e92aec84ac585304e57bd37a40d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36984
I20240502 21:26:59.715938 31388 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:59.716111 31388 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:26:59.719412 31388 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet eb5970b1ec3b4c50b3a59d600fc0696f
I20240502 21:26:59.721053 31647 tablet_service.cc:1514] Processing DeleteTablet for tablet eb5970b1ec3b4c50b3a59d600fc0696f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:59 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:26:59.705513 4832 ts_tablet_manager.cc:1367] T 939c34669f6447f9a2a88da769d0f84f P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.175s user 0.000s sys 0.003s
I20240502 21:26:59.722998 4832 raft_consensus.cc:348] T 939c34669f6447f9a2a88da769d0f84f P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:59.723297 4832 raft_consensus.cc:374] T 939c34669f6447f9a2a88da769d0f84f P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.723469 4832 raft_consensus.cc:729] T 939c34669f6447f9a2a88da769d0f84f P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.723816 4832 consensus_queue.cc:260] T 939c34669f6447f9a2a88da769d0f84f P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:59.724387 4832 ts_tablet_manager.cc:1398] T 939c34669f6447f9a2a88da769d0f84f P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:26:59.724792 4832 tablet_bootstrap.cc:492] T e9140579721c4b3fbba776e4e16e5d9d P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:59.725011 1945 tablet_service.cc:1514] Processing DeleteTablet for tablet eb5970b1ec3b4c50b3a59d600fc0696f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:59 UTC) from {username='slave'} at 127.0.0.1:33002
I20240502 21:26:59.726001 4900 tablet_replica.cc:330] T eb5970b1ec3b4c50b3a59d600fc0696f P 9dec645b473f40578e726f3e50e219bf: stopping tablet replica
I20240502 21:26:59.726325 4900 raft_consensus.cc:2229] T eb5970b1ec3b4c50b3a59d600fc0696f P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:59.726742 4832 tablet_bootstrap.cc:654] T e9140579721c4b3fbba776e4e16e5d9d P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.720516 2077 tablet_service.cc:1514] Processing DeleteTablet for tablet eb5970b1ec3b4c50b3a59d600fc0696f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:26:59 UTC) from {username='slave'} at 127.0.0.1:36740
I20240502 21:26:59.727602 4861 tablet_replica.cc:330] T eb5970b1ec3b4c50b3a59d600fc0696f P 1ade0354c44c4c37ae480c7a94b11e1b: stopping tablet replica
I20240502 21:26:59.727815 4861 raft_consensus.cc:2229] T eb5970b1ec3b4c50b3a59d600fc0696f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:59.728052 4861 raft_consensus.cc:2258] T eb5970b1ec3b4c50b3a59d600fc0696f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:59.728963 4861 ts_tablet_manager.cc:1875] T eb5970b1ec3b4c50b3a59d600fc0696f P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:59.738652 4861 ts_tablet_manager.cc:1888] T eb5970b1ec3b4c50b3a59d600fc0696f P 1ade0354c44c4c37ae480c7a94b11e1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:59.738829 4861 log.cc:1196] T eb5970b1ec3b4c50b3a59d600fc0696f P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal/wals/eb5970b1ec3b4c50b3a59d600fc0696f
I20240502 21:26:59.739358 4861 ts_tablet_manager.cc:1909] T eb5970b1ec3b4c50b3a59d600fc0696f P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting consensus metadata
I20240502 21:26:59.740720 4900 raft_consensus.cc:2258] T eb5970b1ec3b4c50b3a59d600fc0696f P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:59.740792 31381 catalog_manager.cc:4830] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): tablet eb5970b1ec3b4c50b3a59d600fc0696f (table rebalancer_dynamic_table_new_name [id=075b835e0b4942c08a6620c055c49882]) successfully deleted
I20240502 21:26:59.741027 4874 tablet_copy_client.cc:308] T 94ba99a5798f4d5c9442590066896f93 P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.130:37645
I20240502 21:26:59.743445 4883 ts_tablet_manager.cc:1888] T 53b7eec870c44999b48f0bf744b2ffcd P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:26:59.743583 4883 log.cc:1196] T 53b7eec870c44999b48f0bf744b2ffcd P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/53b7eec870c44999b48f0bf744b2ffcd
I20240502 21:26:59.743872 31686 tablet_copy_service.cc:140] P 05f4fac41c4f4959b9b316cb112d0ada: Received BeginTabletCopySession request for tablet 94ba99a5798f4d5c9442590066896f93 from peer 1ade0354c44c4c37ae480c7a94b11e1b ({username='slave'} at 127.24.226.135:46887)
I20240502 21:26:59.744035 31686 tablet_copy_service.cc:161] P 05f4fac41c4f4959b9b316cb112d0ada: Beginning new tablet copy session on tablet 94ba99a5798f4d5c9442590066896f93 from peer 1ade0354c44c4c37ae480c7a94b11e1b at {username='slave'} at 127.24.226.135:46887: session id = 1ade0354c44c4c37ae480c7a94b11e1b-94ba99a5798f4d5c9442590066896f93
I20240502 21:26:59.745121 4900 ts_tablet_manager.cc:1875] T eb5970b1ec3b4c50b3a59d600fc0696f P 9dec645b473f40578e726f3e50e219bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:59.753921 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet 53b7eec870c44999b48f0bf744b2ffcd (table rebalance_test_table_7 [id=0af22f3605164ff58dc3914737c2de4f]) successfully deleted
I20240502 21:26:59.755331 31686 tablet_copy_source_session.cc:205] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:59.765857 4899 tablet_replica.cc:330] T 534b4d96df9b4bc5a9480b6c45eb11a5 P d1406e92aec84ac585304e57bd37a40d: stopping tablet replica
I20240502 21:26:59.766217 4899 raft_consensus.cc:2229] T 534b4d96df9b4bc5a9480b6c45eb11a5 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:59.766602 4899 raft_consensus.cc:2258] T 534b4d96df9b4bc5a9480b6c45eb11a5 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:59.768276 4899 ts_tablet_manager.cc:1875] T 534b4d96df9b4bc5a9480b6c45eb11a5 P d1406e92aec84ac585304e57bd37a40d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:59.768272 4662 raft_consensus.cc:482] T e9140579721c4b3fbba776e4e16e5d9d P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:59.768412 4662 raft_consensus.cc:504] T e9140579721c4b3fbba776e4e16e5d9d P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:59.769410 4662 leader_election.cc:288] T e9140579721c4b3fbba776e4e16e5d9d P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:59.770216 1834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9140579721c4b3fbba776e4e16e5d9d" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
W20240502 21:26:59.770776 2032 leader_election.cc:341] T e9140579721c4b3fbba776e4e16e5d9d P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:59.771135 2225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9140579721c4b3fbba776e4e16e5d9d" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:26:59.776005 2225 raft_consensus.cc:2454] T e9140579721c4b3fbba776e4e16e5d9d P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 0.
I20240502 21:26:59.776423 2032 leader_election.cc:302] T e9140579721c4b3fbba776e4e16e5d9d P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, d1406e92aec84ac585304e57bd37a40d; no voters: 1e04157b98dd4fad98858c290b73c3db
I20240502 21:26:59.776846 4662 raft_consensus.cc:2781] T e9140579721c4b3fbba776e4e16e5d9d P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:59.776943 4662 raft_consensus.cc:482] T e9140579721c4b3fbba776e4e16e5d9d P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:59.777014 4662 raft_consensus.cc:3037] T e9140579721c4b3fbba776e4e16e5d9d P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.777652 4900 ts_tablet_manager.cc:1888] T eb5970b1ec3b4c50b3a59d600fc0696f P 9dec645b473f40578e726f3e50e219bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:59.777755 4900 log.cc:1196] T eb5970b1ec3b4c50b3a59d600fc0696f P 9dec645b473f40578e726f3e50e219bf: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal/wals/eb5970b1ec3b4c50b3a59d600fc0696f
I20240502 21:26:59.778214 4900 ts_tablet_manager.cc:1909] T eb5970b1ec3b4c50b3a59d600fc0696f P 9dec645b473f40578e726f3e50e219bf: Deleting consensus metadata
I20240502 21:26:59.778650 4874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94ba99a5798f4d5c9442590066896f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.779110 31379 catalog_manager.cc:4830] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): tablet eb5970b1ec3b4c50b3a59d600fc0696f (table rebalancer_dynamic_table_new_name [id=075b835e0b4942c08a6620c055c49882]) successfully deleted
I20240502 21:26:59.781143 4662 raft_consensus.cc:504] T e9140579721c4b3fbba776e4e16e5d9d P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:59.782212 1834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9140579721c4b3fbba776e4e16e5d9d" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
W20240502 21:26:59.783828 2032 leader_election.cc:341] T e9140579721c4b3fbba776e4e16e5d9d P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:26:59.784564 2225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9140579721c4b3fbba776e4e16e5d9d" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:26:59.784756 2225 raft_consensus.cc:3037] T e9140579721c4b3fbba776e4e16e5d9d P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.787276 2225 raft_consensus.cc:2454] T e9140579721c4b3fbba776e4e16e5d9d P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 1.
I20240502 21:26:59.787662 2032 leader_election.cc:302] T e9140579721c4b3fbba776e4e16e5d9d P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, d1406e92aec84ac585304e57bd37a40d; no voters: 1e04157b98dd4fad98858c290b73c3db
I20240502 21:26:59.780439 4892 ts_tablet_manager.cc:1888] T e0f96b56fb68439a97cf2aae4aa89b57 P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:26:59.787837 4892 log.cc:1196] T e0f96b56fb68439a97cf2aae4aa89b57 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/e0f96b56fb68439a97cf2aae4aa89b57
I20240502 21:26:59.788741 4662 leader_election.cc:288] T e9140579721c4b3fbba776e4e16e5d9d P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 election: Requested vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:59.788900 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet e0f96b56fb68439a97cf2aae4aa89b57 (table rebalance_test_table_3 [id=59305f5894b34d0085a6cada5a676ca4]) successfully deleted
I20240502 21:26:59.789361 4662 raft_consensus.cc:2781] T e9140579721c4b3fbba776e4e16e5d9d P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:59.788404 4892 tablet_replica.cc:330] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:26:59.790248 4892 raft_consensus.cc:2229] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:26:59.790323 4662 raft_consensus.cc:686] T e9140579721c4b3fbba776e4e16e5d9d P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEADER]: Becoming Leader. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Running, Role: LEADER
I20240502 21:26:59.790570 4662 consensus_queue.cc:237] T e9140579721c4b3fbba776e4e16e5d9d P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:59.790912 4892 raft_consensus.cc:2258] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:59.792802 4892 ts_tablet_manager.cc:1875] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:26:59.797313 4832 tablet_bootstrap.cc:492] T e9140579721c4b3fbba776e4e16e5d9d P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:59.797510 4832 ts_tablet_manager.cc:1367] T e9140579721c4b3fbba776e4e16e5d9d P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.073s user 0.014s sys 0.017s
I20240502 21:26:59.799755 4832 raft_consensus.cc:348] T e9140579721c4b3fbba776e4e16e5d9d P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:59.800004 4832 raft_consensus.cc:374] T e9140579721c4b3fbba776e4e16e5d9d P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.800506 4832 raft_consensus.cc:729] T e9140579721c4b3fbba776e4e16e5d9d P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.800851 4832 consensus_queue.cc:260] T e9140579721c4b3fbba776e4e16e5d9d P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:59.801708 4832 ts_tablet_manager.cc:1398] T e9140579721c4b3fbba776e4e16e5d9d P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 21:26:59.812227 31386 catalog_manager.cc:5484] T e9140579721c4b3fbba776e4e16e5d9d P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135). New cstate: current_term: 1 leader_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:59.819883 4899 ts_tablet_manager.cc:1888] T 534b4d96df9b4bc5a9480b6c45eb11a5 P d1406e92aec84ac585304e57bd37a40d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:26:59.820087 4899 log.cc:1196] T 534b4d96df9b4bc5a9480b6c45eb11a5 P d1406e92aec84ac585304e57bd37a40d: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal/wals/534b4d96df9b4bc5a9480b6c45eb11a5
I20240502 21:26:59.760354 4877 ts_tablet_manager.cc:902] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada: Initiating tablet copy from peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:26:59.820570 4903 ts_tablet_manager.cc:902] T 3e6657723fe84e749dea768de44638e7 P 9dec645b473f40578e726f3e50e219bf: Initiating tablet copy from peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:26:59.820932 31381 catalog_manager.cc:4830] TS d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): tablet 534b4d96df9b4bc5a9480b6c45eb11a5 (table rebalancer_extra_table_20 [id=aae794f4f92e4a4b9f2610d66d67525d]) successfully deleted
I20240502 21:26:59.825870 31386 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:26:59.826431 4892 ts_tablet_manager.cc:1888] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:26:59.826594 4892 log.cc:1196] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/eb5970b1ec3b4c50b3a59d600fc0696f
I20240502 21:26:59.827046 4892 ts_tablet_manager.cc:1909] T eb5970b1ec3b4c50b3a59d600fc0696f P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:26:59.828497 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet eb5970b1ec3b4c50b3a59d600fc0696f (table rebalancer_dynamic_table_new_name [id=075b835e0b4942c08a6620c055c49882]) successfully deleted
I20240502 21:26:59.835958 31775 tablet_service.cc:1467] Processing CreateTablet for tablet fe44f2c6df8a4597a7124ef400d5b3ee (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b143f6f24b724c1cb9b64621ad271585]), partition=
I20240502 21:26:59.836365 31775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe44f2c6df8a4597a7124ef400d5b3ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.836467 1814 tablet_service.cc:1467] Processing CreateTablet for tablet fe44f2c6df8a4597a7124ef400d5b3ee (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b143f6f24b724c1cb9b64621ad271585]), partition=
I20240502 21:26:59.836794 1814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe44f2c6df8a4597a7124ef400d5b3ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.841089 31647 tablet_service.cc:1467] Processing CreateTablet for tablet fe44f2c6df8a4597a7124ef400d5b3ee (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b143f6f24b724c1cb9b64621ad271585]), partition=
I20240502 21:26:59.841379 31647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe44f2c6df8a4597a7124ef400d5b3ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.842773 4832 tablet_bootstrap.cc:492] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:26:59.844615 4832 tablet_bootstrap.cc:654] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.852598 4662 raft_consensus.cc:482] T 939c34669f6447f9a2a88da769d0f84f P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:59.852789 4662 raft_consensus.cc:504] T 939c34669f6447f9a2a88da769d0f84f P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:59.853487 4662 leader_election.cc:288] T 939c34669f6447f9a2a88da769d0f84f P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:26:59.853901 1832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "939c34669f6447f9a2a88da769d0f84f" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:26:59.854141 1832 raft_consensus.cc:2454] T 939c34669f6447f9a2a88da769d0f84f P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 0.
I20240502 21:26:59.855229 2032 leader_election.cc:302] T 939c34669f6447f9a2a88da769d0f84f P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 1e04157b98dd4fad98858c290b73c3db; no voters:
I20240502 21:26:59.855818 4905 raft_consensus.cc:2781] T 939c34669f6447f9a2a88da769d0f84f P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:59.855913 4905 raft_consensus.cc:482] T 939c34669f6447f9a2a88da769d0f84f P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:59.856021 4905 raft_consensus.cc:3037] T 939c34669f6447f9a2a88da769d0f84f P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.856225 4914 tablet_bootstrap.cc:492] T fe44f2c6df8a4597a7124ef400d5b3ee P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:26:59.856280 1965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "939c34669f6447f9a2a88da769d0f84f" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:26:59.856592 1965 raft_consensus.cc:2454] T 939c34669f6447f9a2a88da769d0f84f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 0.
I20240502 21:26:59.859202 4905 raft_consensus.cc:504] T 939c34669f6447f9a2a88da769d0f84f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:59.862061 4914 tablet_bootstrap.cc:654] T fe44f2c6df8a4597a7124ef400d5b3ee P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.863632 1832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "939c34669f6447f9a2a88da769d0f84f" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:26:59.863747 4832 tablet_bootstrap.cc:492] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:26:59.863840 1832 raft_consensus.cc:3037] T 939c34669f6447f9a2a88da769d0f84f P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.863932 4832 ts_tablet_manager.cc:1367] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.006s
I20240502 21:26:59.864723 4832 raft_consensus.cc:348] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:59.864960 4832 raft_consensus.cc:374] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.865046 4832 raft_consensus.cc:729] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.865293 4832 consensus_queue.cc:260] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:59.865865 4832 ts_tablet_manager.cc:1398] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:26:59.866626 1832 raft_consensus.cc:2454] T 939c34669f6447f9a2a88da769d0f84f P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 1.
I20240502 21:26:59.867100 2032 leader_election.cc:302] T 939c34669f6447f9a2a88da769d0f84f P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 1e04157b98dd4fad98858c290b73c3db; no voters:
I20240502 21:26:59.867516 4905 leader_election.cc:288] T 939c34669f6447f9a2a88da769d0f84f P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 election: Requested vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:26:59.867709 4905 raft_consensus.cc:2781] T 939c34669f6447f9a2a88da769d0f84f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:59.867959 4905 raft_consensus.cc:686] T 939c34669f6447f9a2a88da769d0f84f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEADER]: Becoming Leader. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Running, Role: LEADER
I20240502 21:26:59.868259 4905 consensus_queue.cc:237] T 939c34669f6447f9a2a88da769d0f84f P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:26:59.869338 1965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "939c34669f6447f9a2a88da769d0f84f" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf"
I20240502 21:26:59.869554 1965 raft_consensus.cc:3037] T 939c34669f6447f9a2a88da769d0f84f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.870748 4914 tablet_bootstrap.cc:492] T fe44f2c6df8a4597a7124ef400d5b3ee P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:26:59.871393 4914 ts_tablet_manager.cc:1367] T fe44f2c6df8a4597a7124ef400d5b3ee P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20240502 21:26:59.873473 4914 raft_consensus.cc:348] T fe44f2c6df8a4597a7124ef400d5b3ee P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:59.873643 1965 raft_consensus.cc:2454] T 939c34669f6447f9a2a88da769d0f84f P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 1.
I20240502 21:26:59.877113 31392 catalog_manager.cc:5484] T 939c34669f6447f9a2a88da769d0f84f P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135). New cstate: current_term: 1 leader_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:59.878593 4914 raft_consensus.cc:374] T fe44f2c6df8a4597a7124ef400d5b3ee P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.878821 4914 raft_consensus.cc:729] T fe44f2c6df8a4597a7124ef400d5b3ee P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.880584 4914 consensus_queue.cc:260] T fe44f2c6df8a4597a7124ef400d5b3ee P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:59.883312 4914 ts_tablet_manager.cc:1398] T fe44f2c6df8a4597a7124ef400d5b3ee P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 21:26:59.891773 4915 tablet_bootstrap.cc:492] T fe44f2c6df8a4597a7124ef400d5b3ee P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:26:59.896960 4915 tablet_bootstrap.cc:654] T fe44f2c6df8a4597a7124ef400d5b3ee P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:26:59.906078 4874 tablet_copy_client.cc:783] T 94ba99a5798f4d5c9442590066896f93 P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:59.906535 4874 tablet_copy_client.cc:647] T 94ba99a5798f4d5c9442590066896f93 P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:59.911034 4874 tablet_copy_client.cc:523] T 94ba99a5798f4d5c9442590066896f93 P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:59.910894 2225 consensus_queue.cc:237] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } }
I20240502 21:26:59.914557 4874 tablet_bootstrap.cc:492] T 94ba99a5798f4d5c9442590066896f93 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:26:59.917244 4915 tablet_bootstrap.cc:492] T fe44f2c6df8a4597a7124ef400d5b3ee P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
W20240502 21:26:59.917263 2164 consensus_peers.cc:485] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d -> Peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): Couldn't send request to peer 2d50f33557944bb480ea32199d9ee4f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a699f448e116428f85cfee6ccbc98612. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:59.917411 4915 ts_tablet_manager.cc:1367] T fe44f2c6df8a4597a7124ef400d5b3ee P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20240502 21:26:59.917822 31536 raft_consensus.cc:1261] T a699f448e116428f85cfee6ccbc98612 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:59.918380 4499 consensus_queue.cc:1035] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:59.920156 31797 raft_consensus.cc:1261] T a699f448e116428f85cfee6ccbc98612 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:26:59.920758 4915 raft_consensus.cc:348] T fe44f2c6df8a4597a7124ef400d5b3ee P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:59.921554 4903 tablet_copy_client.cc:308] T 3e6657723fe84e749dea768de44638e7 P 9dec645b473f40578e726f3e50e219bf: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.133:39863
I20240502 21:26:59.921525 4499 consensus_queue.cc:1035] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:26:59.921888 4915 raft_consensus.cc:374] T fe44f2c6df8a4597a7124ef400d5b3ee P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:26:59.921960 4915 raft_consensus.cc:729] T fe44f2c6df8a4597a7124ef400d5b3ee P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:26:59.922391 1854 tablet_copy_service.cc:140] P 1e04157b98dd4fad98858c290b73c3db: Received BeginTabletCopySession request for tablet 3e6657723fe84e749dea768de44638e7 from peer 9dec645b473f40578e726f3e50e219bf ({username='slave'} at 127.24.226.134:51927)
I20240502 21:26:59.922546 1854 tablet_copy_service.cc:161] P 1e04157b98dd4fad98858c290b73c3db: Beginning new tablet copy session on tablet 3e6657723fe84e749dea768de44638e7 from peer 9dec645b473f40578e726f3e50e219bf at {username='slave'} at 127.24.226.134:51927: session id = 9dec645b473f40578e726f3e50e219bf-3e6657723fe84e749dea768de44638e7
I20240502 21:26:59.926009 1854 tablet_copy_source_session.cc:205] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:59.922204 4915 consensus_queue.cc:260] T fe44f2c6df8a4597a7124ef400d5b3ee P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:59.927326 4915 ts_tablet_manager.cc:1398] T fe44f2c6df8a4597a7124ef400d5b3ee P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 21:26:59.930507 4903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e6657723fe84e749dea768de44638e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.934664 4499 raft_consensus.cc:2932] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } } }
I20240502 21:26:59.936724 4526 raft_consensus.cc:482] T e9140579721c4b3fbba776e4e16e5d9d P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:59.936897 4526 raft_consensus.cc:504] T e9140579721c4b3fbba776e4e16e5d9d P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:26:59.937376 4877 tablet_copy_client.cc:308] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.134:43753
I20240502 21:26:59.937650 4526 leader_election.cc:288] T e9140579721c4b3fbba776e4e16e5d9d P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:26:59.938086 1986 tablet_copy_service.cc:140] P 9dec645b473f40578e726f3e50e219bf: Received BeginTabletCopySession request for tablet b20c2bb60f6149458a62ee443b8cf3ab from peer 05f4fac41c4f4959b9b316cb112d0ada ({username='slave'} at 127.24.226.130:43265)
I20240502 21:26:59.938256 1986 tablet_copy_service.cc:161] P 9dec645b473f40578e726f3e50e219bf: Beginning new tablet copy session on tablet b20c2bb60f6149458a62ee443b8cf3ab from peer 05f4fac41c4f4959b9b316cb112d0ada at {username='slave'} at 127.24.226.130:43265: session id = 05f4fac41c4f4959b9b316cb112d0ada-b20c2bb60f6149458a62ee443b8cf3ab
I20240502 21:26:59.938398 2093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9140579721c4b3fbba776e4e16e5d9d" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:26:59.938452 31796 raft_consensus.cc:2932] T a699f448e116428f85cfee6ccbc98612 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } } }
I20240502 21:26:59.939766 31533 raft_consensus.cc:2932] T a699f448e116428f85cfee6ccbc98612 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } } }
I20240502 21:26:59.941190 1986 tablet_copy_source_session.cc:205] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:59.942705 2225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9140579721c4b3fbba776e4e16e5d9d" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:26:59.943017 2225 raft_consensus.cc:2379] T e9140579721c4b3fbba776e4e16e5d9d P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e04157b98dd4fad98858c290b73c3db in current term 1: Already voted for candidate 1ade0354c44c4c37ae480c7a94b11e1b in this term.
I20240502 21:26:59.943470 1769 leader_election.cc:302] T e9140579721c4b3fbba776e4e16e5d9d P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db; no voters: 1ade0354c44c4c37ae480c7a94b11e1b, d1406e92aec84ac585304e57bd37a40d
I20240502 21:26:59.943953 4526 raft_consensus.cc:3037] T e9140579721c4b3fbba776e4e16e5d9d P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.946563 4526 raft_consensus.cc:2726] T e9140579721c4b3fbba776e4e16e5d9d P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:26:59.946357 31392 catalog_manager.cc:5484] T a699f448e116428f85cfee6ccbc98612 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index -1 to 2, NON_VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) added. New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } } }
I20240502 21:26:59.947338 4877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b20c2bb60f6149458a62ee443b8cf3ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:59.950095 4526 raft_consensus.cc:482] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:26:59.950235 4526 raft_consensus.cc:504] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:59.950914 4526 leader_election.cc:288] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:59.951550 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe44f2c6df8a4597a7124ef400d5b3ee" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:26:59.951735 31665 raft_consensus.cc:2454] T fe44f2c6df8a4597a7124ef400d5b3ee P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 0.
I20240502 21:26:59.952579 1770 leader_election.cc:302] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 1e04157b98dd4fad98858c290b73c3db; no voters:
I20240502 21:26:59.952858 4526 raft_consensus.cc:2781] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:26:59.952977 4526 raft_consensus.cc:482] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:26:59.953078 4526 raft_consensus.cc:3037] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.952800 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe44f2c6df8a4597a7124ef400d5b3ee" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:26:59.953460 31797 raft_consensus.cc:2454] T fe44f2c6df8a4597a7124ef400d5b3ee P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 0.
I20240502 21:26:59.955456 4526 raft_consensus.cc:504] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:59.956936 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe44f2c6df8a4597a7124ef400d5b3ee" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:26:59.957123 31665 raft_consensus.cc:3037] T fe44f2c6df8a4597a7124ef400d5b3ee P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.959733 31665 raft_consensus.cc:2454] T fe44f2c6df8a4597a7124ef400d5b3ee P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:26:59.960815 1770 leader_election.cc:302] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 1e04157b98dd4fad98858c290b73c3db; no voters:
I20240502 21:26:59.961114 4744 raft_consensus.cc:2781] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:26:59.961261 4744 raft_consensus.cc:686] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Becoming Leader. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Running, Role: LEADER
I20240502 21:26:59.961572 4744 consensus_queue.cc:237] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:26:59.965721 3818 consensus_queue.cc:1035] T a8684b1ca3a642688a232afaf1b28f96 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:59.966022 4526 leader_election.cc:288] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Requested vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:26:59.966117 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe44f2c6df8a4597a7124ef400d5b3ee" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:26:59.967581 31797 raft_consensus.cc:3037] T fe44f2c6df8a4597a7124ef400d5b3ee P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:26:59.972321 4801 consensus_queue.cc:1035] T a8684b1ca3a642688a232afaf1b28f96 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:26:59.979889 31797 raft_consensus.cc:2454] T fe44f2c6df8a4597a7124ef400d5b3ee P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:26:59.982254 4903 tablet_copy_client.cc:783] T 3e6657723fe84e749dea768de44638e7 P 9dec645b473f40578e726f3e50e219bf: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:59.985141 31386 catalog_manager.cc:5484] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e04157b98dd4fad98858c290b73c3db (127.24.226.133). New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:59.991509 4877 tablet_copy_client.cc:783] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:59.991770 4877 tablet_copy_client.cc:647] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:59.994490 4903 tablet_copy_client.cc:647] T 3e6657723fe84e749dea768de44638e7 P 9dec645b473f40578e726f3e50e219bf: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:59.998476 4877 tablet_copy_client.cc:523] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:00.005381 4903 tablet_copy_client.cc:523] T 3e6657723fe84e749dea768de44638e7 P 9dec645b473f40578e726f3e50e219bf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:00.006338 4877 tablet_bootstrap.cc:492] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:27:00.014863 4903 tablet_bootstrap.cc:492] T 3e6657723fe84e749dea768de44638e7 P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:27:00.032757 4903 tablet_bootstrap.cc:492] T 3e6657723fe84e749dea768de44638e7 P 9dec645b473f40578e726f3e50e219bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:00.033252 4903 tablet_bootstrap.cc:492] T 3e6657723fe84e749dea768de44638e7 P 9dec645b473f40578e726f3e50e219bf: Bootstrap complete.
I20240502 21:27:00.033594 4903 ts_tablet_manager.cc:1367] T 3e6657723fe84e749dea768de44638e7 P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20240502 21:27:00.034430 4903 raft_consensus.cc:348] T 3e6657723fe84e749dea768de44638e7 P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:27:00.034745 4903 raft_consensus.cc:729] T 3e6657723fe84e749dea768de44638e7 P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: LEARNER
I20240502 21:27:00.035017 4903 consensus_queue.cc:260] T 3e6657723fe84e749dea768de44638e7 P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:27:00.035790 4903 ts_tablet_manager.cc:1398] T 3e6657723fe84e749dea768de44638e7 P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:00.040035 1854 tablet_copy_service.cc:342] P 1e04157b98dd4fad98858c290b73c3db: Request end of tablet copy session 9dec645b473f40578e726f3e50e219bf-3e6657723fe84e749dea768de44638e7 received from {username='slave'} at 127.24.226.134:51927
I20240502 21:27:00.040207 1854 tablet_copy_service.cc:433] P 1e04157b98dd4fad98858c290b73c3db: ending tablet copy session 9dec645b473f40578e726f3e50e219bf-3e6657723fe84e749dea768de44638e7 on tablet 3e6657723fe84e749dea768de44638e7 with peer 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:00.059540 31536 consensus_queue.cc:237] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } }
I20240502 21:27:00.062273 4877 tablet_bootstrap.cc:492] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:00.062706 4877 tablet_bootstrap.cc:492] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap complete.
I20240502 21:27:00.063341 4877 ts_tablet_manager.cc:1367] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.057s user 0.023s sys 0.021s
I20240502 21:27:00.064033 4877 raft_consensus.cc:348] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } }
I20240502 21:27:00.064658 4877 raft_consensus.cc:729] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: LEARNER
I20240502 21:27:00.064878 4877 consensus_queue.cc:260] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } }
I20240502 21:27:00.065667 4874 tablet_bootstrap.cc:492] T 94ba99a5798f4d5c9442590066896f93 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:00.066862 4874 tablet_bootstrap.cc:492] T 94ba99a5798f4d5c9442590066896f93 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap complete.
I20240502 21:27:00.067204 4874 ts_tablet_manager.cc:1367] T 94ba99a5798f4d5c9442590066896f93 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.153s user 0.041s sys 0.008s
I20240502 21:27:00.068090 4874 raft_consensus.cc:348] T 94ba99a5798f4d5c9442590066896f93 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } }
I20240502 21:27:00.068506 4874 raft_consensus.cc:729] T 94ba99a5798f4d5c9442590066896f93 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: LEARNER
I20240502 21:27:00.068605 4877 ts_tablet_manager.cc:1398] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:27:00.068779 4874 consensus_queue.cc:260] T 94ba99a5798f4d5c9442590066896f93 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } }
W20240502 21:27:00.069105 31471 consensus_peers.cc:485] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f -> Peer 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): Couldn't send request to peer 1ade0354c44c4c37ae480c7a94b11e1b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd0198178da640688230aeb5a1106cb4. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:00.069314 1986 tablet_copy_service.cc:342] P 9dec645b473f40578e726f3e50e219bf: Request end of tablet copy session 05f4fac41c4f4959b9b316cb112d0ada-b20c2bb60f6149458a62ee443b8cf3ab received from {username='slave'} at 127.24.226.130:43265
I20240502 21:27:00.069494 1986 tablet_copy_service.cc:433] P 9dec645b473f40578e726f3e50e219bf: ending tablet copy session 05f4fac41c4f4959b9b316cb112d0ada-b20c2bb60f6149458a62ee443b8cf3ab on tablet b20c2bb60f6149458a62ee443b8cf3ab with peer 05f4fac41c4f4959b9b316cb112d0ada
I20240502 21:27:00.070276 31797 raft_consensus.cc:1261] T dd0198178da640688230aeb5a1106cb4 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:00.070809 4430 consensus_queue.cc:1035] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:00.071780 31925 raft_consensus.cc:1261] T dd0198178da640688230aeb5a1106cb4 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:00.073305 4874 ts_tablet_manager.cc:1398] T 94ba99a5798f4d5c9442590066896f93 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 21:27:00.075155 4430 consensus_queue.cc:1035] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:00.075536 31686 tablet_copy_service.cc:342] P 05f4fac41c4f4959b9b316cb112d0ada: Request end of tablet copy session 1ade0354c44c4c37ae480c7a94b11e1b-94ba99a5798f4d5c9442590066896f93 received from {username='slave'} at 127.24.226.135:46887
I20240502 21:27:00.075621 31686 tablet_copy_service.cc:433] P 05f4fac41c4f4959b9b316cb112d0ada: ending tablet copy session 1ade0354c44c4c37ae480c7a94b11e1b-94ba99a5798f4d5c9442590066896f93 on tablet 94ba99a5798f4d5c9442590066896f93 with peer 1ade0354c44c4c37ae480c7a94b11e1b
I20240502 21:27:00.078361 4938 raft_consensus.cc:2932] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } } }
I20240502 21:27:00.083117 31925 raft_consensus.cc:2932] T dd0198178da640688230aeb5a1106cb4 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } } }
I20240502 21:27:00.084381 31392 catalog_manager.cc:5484] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:00.091583 31386 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:00.097920 31796 raft_consensus.cc:1261] T fe44f2c6df8a4597a7124ef400d5b3ee P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:00.098565 31665 raft_consensus.cc:1261] T fe44f2c6df8a4597a7124ef400d5b3ee P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:00.098537 4526 consensus_queue.cc:1035] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.100071 4526 consensus_queue.cc:1035] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.104200 31797 raft_consensus.cc:2932] T dd0198178da640688230aeb5a1106cb4 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } } }
I20240502 21:27:00.108594 4499 raft_consensus.cc:1050] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d: attempting to promote NON_VOTER 08d7a0680d244b55b93d17f9ab57b84c to VOTER
I20240502 21:27:00.109717 4499 consensus_queue.cc:237] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } }
I20240502 21:27:00.120144 31386 catalog_manager.cc:5576] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807) does not have the latest schema for tablet fe44f2c6df8a4597a7124ef400d5b3ee (table rebalancer_dynamic_table [id=b143f6f24b724c1cb9b64621ad271585]). Expected version 1 got 0
I20240502 21:27:00.121064 4954 tablet.cc:1722] T fe44f2c6df8a4597a7124ef400d5b3ee P 08d7a0680d244b55b93d17f9ab57b84c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:00.122543 4954 tablet.cc:1620] T fe44f2c6df8a4597a7124ef400d5b3ee P 08d7a0680d244b55b93d17f9ab57b84c: MemRowSet was empty: no flush needed.
I20240502 21:27:00.129823 31386 catalog_manager.cc:5576] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645) does not have the latest schema for tablet fe44f2c6df8a4597a7124ef400d5b3ee (table rebalancer_dynamic_table [id=b143f6f24b724c1cb9b64621ad271585]). Expected version 1 got 0
I20240502 21:27:00.130262 31392 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=b143f6f24b724c1cb9b64621ad271585] alter complete (version 1)
I20240502 21:27:00.131611 4953 tablet.cc:1722] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:00.133319 4953 tablet.cc:1620] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db: MemRowSet was empty: no flush needed.
I20240502 21:27:00.136730 31392 catalog_manager.cc:5576] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863) does not have the latest schema for tablet fe44f2c6df8a4597a7124ef400d5b3ee (table rebalancer_dynamic_table [id=b143f6f24b724c1cb9b64621ad271585]). Expected version 1 got 0
I20240502 21:27:00.140717 31796 raft_consensus.cc:1261] T c5ba5610b754466fb1e12db451295809 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEARNER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:00.141530 4498 consensus_queue.cc:1035] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:00.148703 31925 raft_consensus.cc:1261] T c5ba5610b754466fb1e12db451295809 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:00.151597 4407 consensus_queue.cc:1035] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:00.152535 1965 raft_consensus.cc:1261] T c5ba5610b754466fb1e12db451295809 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:00.153193 4407 consensus_queue.cc:1035] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:00.169548 31923 consensus_queue.cc:237] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } }
I20240502 21:27:00.175127 31667 raft_consensus.cc:1261] T ae56d18c629a4e5eb50911c89de4227c P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:00.176151 4822 consensus_queue.cc:1035] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:00.178393 1964 raft_consensus.cc:1261] T ae56d18c629a4e5eb50911c89de4227c P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 21:27:00.179097 31862 consensus_peers.cc:485] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 -> Peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): Couldn't send request to peer d1406e92aec84ac585304e57bd37a40d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae56d18c629a4e5eb50911c89de4227c. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:00.179509 4723 consensus_queue.cc:1035] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:00.192154 4956 tablet.cc:1722] T fe44f2c6df8a4597a7124ef400d5b3ee P 05f4fac41c4f4959b9b316cb112d0ada: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:00.192682 4956 tablet.cc:1620] T fe44f2c6df8a4597a7124ef400d5b3ee P 05f4fac41c4f4959b9b316cb112d0ada: MemRowSet was empty: no flush needed.
I20240502 21:27:00.195952 31392 catalog_manager.cc:5576] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645) does not have the latest schema for tablet fe44f2c6df8a4597a7124ef400d5b3ee (table rebalancer_dynamic_table [id=b143f6f24b724c1cb9b64621ad271585]). Expected version 1 got 0
I20240502 21:27:00.201551 4499 raft_consensus.cc:2932] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:27:00.209885 4963 raft_consensus.cc:2932] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } } }
I20240502 21:27:00.210520 31923 raft_consensus.cc:2932] T c5ba5610b754466fb1e12db451295809 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:27:00.213716 31665 raft_consensus.cc:2932] T ae56d18c629a4e5eb50911c89de4227c P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } } }
I20240502 21:27:00.215674 1964 raft_consensus.cc:2932] T c5ba5610b754466fb1e12db451295809 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:27:00.222740 31386 catalog_manager.cc:5484] T c5ba5610b754466fb1e12db451295809 P 9dec645b473f40578e726f3e50e219bf reported cstate change: config changed from index 2 to 3, 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:27:00.224242 31393 catalog_manager.cc:5484] T ae56d18c629a4e5eb50911c89de4227c P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: config changed from index 6 to 7, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } } }
I20240502 21:27:00.228578 31796 raft_consensus.cc:2932] T c5ba5610b754466fb1e12db451295809 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:27:00.249747 31386 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:00.250458 1965 raft_consensus.cc:2932] T ae56d18c629a4e5eb50911c89de4227c P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } } }
I20240502 21:27:00.256140 2093 raft_consensus.cc:1203] T 94ba99a5798f4d5c9442590066896f93 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:00.266144 2225 consensus_queue.cc:237] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } }
I20240502 21:27:00.268256 31796 raft_consensus.cc:1261] T c5ba5610b754466fb1e12db451295809 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:00.268289 31925 raft_consensus.cc:1261] T c5ba5610b754466fb1e12db451295809 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:00.268752 4498 consensus_queue.cc:1035] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:00.269279 4407 consensus_queue.cc:1035] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:00.269477 4662 consensus_queue.cc:1035] T 939c34669f6447f9a2a88da769d0f84f P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.276887 4407 raft_consensus.cc:2932] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:27:00.278934 4953 tablet.cc:1722] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:00.279491 4953 tablet.cc:1620] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db: MemRowSet was empty: no flush needed.
I20240502 21:27:00.281040 4956 tablet.cc:1722] T fe44f2c6df8a4597a7124ef400d5b3ee P 05f4fac41c4f4959b9b316cb112d0ada: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:00.281579 4956 tablet.cc:1620] T fe44f2c6df8a4597a7124ef400d5b3ee P 05f4fac41c4f4959b9b316cb112d0ada: MemRowSet was empty: no flush needed.
I20240502 21:27:00.283062 31381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c5ba5610b754466fb1e12db451295809 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:00.282863 31925 raft_consensus.cc:2932] T c5ba5610b754466fb1e12db451295809 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:27:00.290807 31389 catalog_manager.cc:5484] T c5ba5610b754466fb1e12db451295809 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: config changed from index 3 to 4, VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) evicted. New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:27:00.292033 4954 tablet.cc:1722] T fe44f2c6df8a4597a7124ef400d5b3ee P 08d7a0680d244b55b93d17f9ab57b84c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:00.292470 4954 tablet.cc:1620] T fe44f2c6df8a4597a7124ef400d5b3ee P 08d7a0680d244b55b93d17f9ab57b84c: MemRowSet was empty: no flush needed.
I20240502 21:27:00.298467 1945 tablet_service.cc:1514] Processing DeleteTablet for tablet c5ba5610b754466fb1e12db451295809 with delete_type TABLET_DATA_TOMBSTONED (TS 9dec645b473f40578e726f3e50e219bf not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33002
I20240502 21:27:00.301823 31392 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=b143f6f24b724c1cb9b64621ad271585] alter complete (version 2)
I20240502 21:27:00.304204 4905 consensus_queue.cc:1035] T 939c34669f6447f9a2a88da769d0f84f P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.302435 31796 raft_consensus.cc:2932] T c5ba5610b754466fb1e12db451295809 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:27:00.310195 4905 consensus_queue.cc:1035] T e9140579721c4b3fbba776e4e16e5d9d P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.311055 1965 raft_consensus.cc:1203] T 3e6657723fe84e749dea768de44638e7 P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:00.320933 4974 tablet_replica.cc:330] T c5ba5610b754466fb1e12db451295809 P 9dec645b473f40578e726f3e50e219bf: stopping tablet replica
I20240502 21:27:00.321163 4974 raft_consensus.cc:2229] T c5ba5610b754466fb1e12db451295809 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:00.321768 4974 raft_consensus.cc:2258] T c5ba5610b754466fb1e12db451295809 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:00.324149 4974 ts_tablet_manager.cc:1875] T c5ba5610b754466fb1e12db451295809 P 9dec645b473f40578e726f3e50e219bf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:00.330384 4974 ts_tablet_manager.cc:1888] T c5ba5610b754466fb1e12db451295809 P 9dec645b473f40578e726f3e50e219bf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:27:00.330530 4974 log.cc:1196] T c5ba5610b754466fb1e12db451295809 P 9dec645b473f40578e726f3e50e219bf: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal/wals/c5ba5610b754466fb1e12db451295809
I20240502 21:27:00.330873 4905 consensus_queue.cc:1035] T e9140579721c4b3fbba776e4e16e5d9d P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:00.333117 31379 catalog_manager.cc:4830] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): tablet c5ba5610b754466fb1e12db451295809 (table rebalancer_extra_table_10 [id=112fc5f7af4e4c8080926e0a6128b468]) successfully deleted
I20240502 21:27:00.375969 1832 consensus_queue.cc:237] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } }
I20240502 21:27:00.378201 2093 raft_consensus.cc:1261] T 9bc57b8a0388416290997df544e5cc0b P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:00.378221 1964 raft_consensus.cc:1261] T 9bc57b8a0388416290997df544e5cc0b P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:27:00.378434 1769 consensus_peers.cc:485] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db -> Peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): Couldn't send request to peer 08d7a0680d244b55b93d17f9ab57b84c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9bc57b8a0388416290997df544e5cc0b. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:00.378818 4526 consensus_queue.cc:1035] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:00.379103 4744 consensus_queue.cc:1035] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:00.381865 4336 raft_consensus.cc:2932] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } } }
I20240502 21:27:00.382387 1964 raft_consensus.cc:2932] T 9bc57b8a0388416290997df544e5cc0b P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } } }
I20240502 21:27:00.385943 2093 raft_consensus.cc:2932] T 9bc57b8a0388416290997df544e5cc0b P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } } }
I20240502 21:27:00.388110 31393 catalog_manager.cc:5484] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db reported cstate change: config changed from index -1 to 2, NON_VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) added. New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:00.413802 2230 consensus_queue.cc:237] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:27:00.415999 31923 raft_consensus.cc:1261] T 59f36d947f4341f39b6969cc1e065b6c P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:27:00.416334 2161 consensus_peers.cc:485] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 59f36d947f4341f39b6969cc1e065b6c. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:00.416555 2093 raft_consensus.cc:1261] T 59f36d947f4341f39b6969cc1e065b6c P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:00.416606 4966 consensus_queue.cc:1035] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:00.417061 4499 consensus_queue.cc:1035] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:00.419126 4499 raft_consensus.cc:2932] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:27:00.419951 31923 raft_consensus.cc:2932] T 59f36d947f4341f39b6969cc1e065b6c P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:27:00.420692 2093 raft_consensus.cc:2932] T 59f36d947f4341f39b6969cc1e065b6c P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:27:00.425653 31389 catalog_manager.cc:5484] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d reported cstate change: config changed from index -1 to 2, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:00.426892 31392 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:00.435648 4953 tablet.cc:1722] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:00.436862 4954 tablet.cc:1722] T fe44f2c6df8a4597a7124ef400d5b3ee P 08d7a0680d244b55b93d17f9ab57b84c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:00.439118 4956 tablet.cc:1722] T fe44f2c6df8a4597a7124ef400d5b3ee P 05f4fac41c4f4959b9b316cb112d0ada: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:00.446492 31386 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table_new_name [id=b143f6f24b724c1cb9b64621ad271585] alter complete (version 3)
I20240502 21:27:00.447319 4996 ts_tablet_manager.cc:902] T a699f448e116428f85cfee6ccbc98612 P 2d50f33557944bb480ea32199d9ee4f1: Initiating tablet copy from peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:27:00.451037 4996 tablet_copy_client.cc:308] T a699f448e116428f85cfee6ccbc98612 P 2d50f33557944bb480ea32199d9ee4f1: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.136:34907
I20240502 21:27:00.451862 2251 tablet_copy_service.cc:140] P d1406e92aec84ac585304e57bd37a40d: Received BeginTabletCopySession request for tablet a699f448e116428f85cfee6ccbc98612 from peer 2d50f33557944bb480ea32199d9ee4f1 ({username='slave'} at 127.24.226.132:59119)
I20240502 21:27:00.452008 2251 tablet_copy_service.cc:161] P d1406e92aec84ac585304e57bd37a40d: Beginning new tablet copy session on tablet a699f448e116428f85cfee6ccbc98612 from peer 2d50f33557944bb480ea32199d9ee4f1 at {username='slave'} at 127.24.226.132:59119: session id = 2d50f33557944bb480ea32199d9ee4f1-a699f448e116428f85cfee6ccbc98612
I20240502 21:27:00.457136 2251 tablet_copy_source_session.cc:205] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:00.458364 4996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a699f448e116428f85cfee6ccbc98612. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.463202 4996 tablet_copy_client.cc:783] T a699f448e116428f85cfee6ccbc98612 P 2d50f33557944bb480ea32199d9ee4f1: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:00.463490 4996 tablet_copy_client.cc:647] T a699f448e116428f85cfee6ccbc98612 P 2d50f33557944bb480ea32199d9ee4f1: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:00.464879 4996 tablet_copy_client.cc:523] T a699f448e116428f85cfee6ccbc98612 P 2d50f33557944bb480ea32199d9ee4f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:00.467320 4996 tablet_bootstrap.cc:492] T a699f448e116428f85cfee6ccbc98612 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:27:00.467253 1964 consensus_queue.cc:237] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } }
I20240502 21:27:00.469708 2230 raft_consensus.cc:1261] T 51551a1967b74094b373cc030e2bf850 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:00.469962 31667 raft_consensus.cc:1261] T 51551a1967b74094b373cc030e2bf850 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:00.470279 3818 consensus_queue.cc:1035] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:27:00.470614 1902 consensus_peers.cc:485] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf -> Peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Couldn't send request to peer a44f41b0b6644fb6aa29476aed8c964f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 51551a1967b74094b373cc030e2bf850. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:00.470921 3818 consensus_queue.cc:1035] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:00.472506 4996 tablet_bootstrap.cc:492] T a699f448e116428f85cfee6ccbc98612 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:00.472934 4996 tablet_bootstrap.cc:492] T a699f448e116428f85cfee6ccbc98612 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap complete.
I20240502 21:27:00.473146 4996 ts_tablet_manager.cc:1367] T a699f448e116428f85cfee6ccbc98612 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.006s
I20240502 21:27:00.472862 4801 raft_consensus.cc:2932] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } } }
I20240502 21:27:00.473359 2230 raft_consensus.cc:2932] T 51551a1967b74094b373cc030e2bf850 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } } }
I20240502 21:27:00.475409 4996 raft_consensus.cc:348] T a699f448e116428f85cfee6ccbc98612 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } }
I20240502 21:27:00.475296 31667 raft_consensus.cc:2932] T 51551a1967b74094b373cc030e2bf850 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } } }
I20240502 21:27:00.475666 4996 raft_consensus.cc:729] T a699f448e116428f85cfee6ccbc98612 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: LEARNER
I20240502 21:27:00.475838 4996 consensus_queue.cc:260] T a699f448e116428f85cfee6ccbc98612 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } }
I20240502 21:27:00.477810 4996 ts_tablet_manager.cc:1398] T a699f448e116428f85cfee6ccbc98612 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:27:00.478981 31386 catalog_manager.cc:5484] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf reported cstate change: config changed from index -1 to 2, NON_VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) added. New cstate: current_term: 1 leader_uuid: "9dec645b473f40578e726f3e50e219bf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:00.479729 2251 tablet_copy_service.cc:342] P d1406e92aec84ac585304e57bd37a40d: Request end of tablet copy session 2d50f33557944bb480ea32199d9ee4f1-a699f448e116428f85cfee6ccbc98612 received from {username='slave'} at 127.24.226.132:59119
I20240502 21:27:00.479861 2251 tablet_copy_service.cc:433] P d1406e92aec84ac585304e57bd37a40d: ending tablet copy session 2d50f33557944bb480ea32199d9ee4f1-a699f448e116428f85cfee6ccbc98612 on tablet a699f448e116428f85cfee6ccbc98612 with peer 2d50f33557944bb480ea32199d9ee4f1
I20240502 21:27:00.499706 2093 consensus_queue.cc:237] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } }
I20240502 21:27:00.501688 2230 raft_consensus.cc:1261] T f74ea10ab8a2459794f816707b87f494 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 1ade0354c44c4c37ae480c7a94b11e1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:27:00.501968 2032 consensus_peers.cc:485] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): Couldn't send request to peer 08d7a0680d244b55b93d17f9ab57b84c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f74ea10ab8a2459794f816707b87f494. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:00.502153 1832 raft_consensus.cc:1261] T f74ea10ab8a2459794f816707b87f494 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer 1ade0354c44c4c37ae480c7a94b11e1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:00.502209 4905 consensus_queue.cc:1035] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:00.502687 4905 consensus_queue.cc:1035] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:00.504717 4904 raft_consensus.cc:2932] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } } }
I20240502 21:27:00.505265 2230 raft_consensus.cc:2932] T f74ea10ab8a2459794f816707b87f494 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } } }
I20240502 21:27:00.505735 1832 raft_consensus.cc:2932] T f74ea10ab8a2459794f816707b87f494 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } } }
I20240502 21:27:00.510892 31389 catalog_manager.cc:5484] T f74ea10ab8a2459794f816707b87f494 P d1406e92aec84ac585304e57bd37a40d reported cstate change: config changed from index -1 to 2, NON_VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) added. New cstate: current_term: 1 leader_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } } }
I20240502 21:27:00.517218 31392 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_extra_table_34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:00.526396 1814 tablet_service.cc:1467] Processing CreateTablet for tablet 18058b4a7c434ec8aff7832fad7ab7b7 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ff68087b16a74afc8331c1462b2d3ae1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:00.526795 1814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18058b4a7c434ec8aff7832fad7ab7b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.527309 31514 tablet_service.cc:1467] Processing CreateTablet for tablet a8328088928f4de19f7cf20189382d91 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ff68087b16a74afc8331c1462b2d3ae1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:00.527674 31514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8328088928f4de19f7cf20189382d91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.528281 2210 tablet_service.cc:1467] Processing CreateTablet for tablet 18058b4a7c434ec8aff7832fad7ab7b7 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ff68087b16a74afc8331c1462b2d3ae1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:00.528673 2210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18058b4a7c434ec8aff7832fad7ab7b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.528812 2208 tablet_service.cc:1467] Processing CreateTablet for tablet a8328088928f4de19f7cf20189382d91 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ff68087b16a74afc8331c1462b2d3ae1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:00.529192 2208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8328088928f4de19f7cf20189382d91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.531599 1945 tablet_service.cc:1467] Processing CreateTablet for tablet a8328088928f4de19f7cf20189382d91 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ff68087b16a74afc8331c1462b2d3ae1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:00.532019 1945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8328088928f4de19f7cf20189382d91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.533006 31775 tablet_service.cc:1467] Processing CreateTablet for tablet b67096e38f2e4c42a33e74f0145a04cc (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ff68087b16a74afc8331c1462b2d3ae1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:00.533397 31775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b67096e38f2e4c42a33e74f0145a04cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.533488 1813 tablet_service.cc:1467] Processing CreateTablet for tablet b67096e38f2e4c42a33e74f0145a04cc (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ff68087b16a74afc8331c1462b2d3ae1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:00.533833 1813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b67096e38f2e4c42a33e74f0145a04cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.537096 1946 tablet_service.cc:1467] Processing CreateTablet for tablet b67096e38f2e4c42a33e74f0145a04cc (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ff68087b16a74afc8331c1462b2d3ae1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:00.537503 1946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b67096e38f2e4c42a33e74f0145a04cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.537521 2077 tablet_service.cc:1467] Processing CreateTablet for tablet 18058b4a7c434ec8aff7832fad7ab7b7 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ff68087b16a74afc8331c1462b2d3ae1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:00.537901 2077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18058b4a7c434ec8aff7832fad7ab7b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.548928 4874 ts_tablet_manager.cc:902] T dd0198178da640688230aeb5a1106cb4 P 1ade0354c44c4c37ae480c7a94b11e1b: Initiating tablet copy from peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:27:00.550530 5029 tablet_bootstrap.cc:492] T a8328088928f4de19f7cf20189382d91 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:27:00.552510 31392 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:00.552754 31392 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:00.556810 5032 tablet_bootstrap.cc:492] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:27:00.558557 5032 tablet_bootstrap.cc:654] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:00.561725 31392 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet fe44f2c6df8a4597a7124ef400d5b3ee
I20240502 21:27:00.562300 31647 tablet_service.cc:1514] Processing DeleteTablet for tablet fe44f2c6df8a4597a7124ef400d5b3ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:00 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:27:00.562901 1813 tablet_service.cc:1514] Processing DeleteTablet for tablet fe44f2c6df8a4597a7124ef400d5b3ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:00 UTC) from {username='slave'} at 127.0.0.1:47510
I20240502 21:27:00.563647 5029 tablet_bootstrap.cc:654] T a8328088928f4de19f7cf20189382d91 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:00.567741 5027 tablet_bootstrap.cc:492] T a8328088928f4de19f7cf20189382d91 P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:27:00.568888 5029 tablet_bootstrap.cc:492] T a8328088928f4de19f7cf20189382d91 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:27:00.570909 5029 ts_tablet_manager.cc:1367] T a8328088928f4de19f7cf20189382d91 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.001s
I20240502 21:27:00.571374 5032 tablet_bootstrap.cc:492] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:27:00.571570 5032 ts_tablet_manager.cc:1367] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.005s
I20240502 21:27:00.571812 31774 tablet_service.cc:1514] Processing DeleteTablet for tablet fe44f2c6df8a4597a7124ef400d5b3ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:00 UTC) from {username='slave'} at 127.0.0.1:51596
I20240502 21:27:00.572583 5032 raft_consensus.cc:348] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:00.572847 5032 raft_consensus.cc:374] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:00.572993 5032 raft_consensus.cc:729] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.572999 5029 raft_consensus.cc:348] T a8328088928f4de19f7cf20189382d91 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:00.573348 5032 consensus_queue.cc:260] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:00.573912 5032 ts_tablet_manager.cc:1398] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:00.574301 5039 tablet_replica.cc:330] T fe44f2c6df8a4597a7124ef400d5b3ee P 08d7a0680d244b55b93d17f9ab57b84c: stopping tablet replica
I20240502 21:27:00.574510 5039 raft_consensus.cc:2229] T fe44f2c6df8a4597a7124ef400d5b3ee P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:00.574671 5039 raft_consensus.cc:2258] T fe44f2c6df8a4597a7124ef400d5b3ee P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:00.575459 5027 tablet_bootstrap.cc:654] T a8328088928f4de19f7cf20189382d91 P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:00.575686 5039 ts_tablet_manager.cc:1875] T fe44f2c6df8a4597a7124ef400d5b3ee P 08d7a0680d244b55b93d17f9ab57b84c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:00.575870 5029 raft_consensus.cc:374] T a8328088928f4de19f7cf20189382d91 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:00.576120 5029 raft_consensus.cc:729] T a8328088928f4de19f7cf20189382d91 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.576524 5029 consensus_queue.cc:260] T a8328088928f4de19f7cf20189382d91 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:00.578372 5029 ts_tablet_manager.cc:1398] T a8328088928f4de19f7cf20189382d91 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240502 21:27:00.579138 5029 tablet_bootstrap.cc:492] T 18058b4a7c434ec8aff7832fad7ab7b7 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:27:00.582499 5039 ts_tablet_manager.cc:1888] T fe44f2c6df8a4597a7124ef400d5b3ee P 08d7a0680d244b55b93d17f9ab57b84c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:00.582880 5039 log.cc:1196] T fe44f2c6df8a4597a7124ef400d5b3ee P 08d7a0680d244b55b93d17f9ab57b84c: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/wals/fe44f2c6df8a4597a7124ef400d5b3ee
I20240502 21:27:00.583348 5039 ts_tablet_manager.cc:1909] T fe44f2c6df8a4597a7124ef400d5b3ee P 08d7a0680d244b55b93d17f9ab57b84c: Deleting consensus metadata
I20240502 21:27:00.584560 31381 catalog_manager.cc:4830] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): tablet fe44f2c6df8a4597a7124ef400d5b3ee (table rebalancer_dynamic_table_new_name [id=b143f6f24b724c1cb9b64621ad271585]) successfully deleted
I20240502 21:27:00.585038 5029 tablet_bootstrap.cc:654] T 18058b4a7c434ec8aff7832fad7ab7b7 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:00.585733 5027 tablet_bootstrap.cc:492] T a8328088928f4de19f7cf20189382d91 P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:27:00.585860 5027 ts_tablet_manager.cc:1367] T a8328088928f4de19f7cf20189382d91 P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240502 21:27:00.586357 5027 raft_consensus.cc:348] T a8328088928f4de19f7cf20189382d91 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:00.586567 5027 raft_consensus.cc:374] T a8328088928f4de19f7cf20189382d91 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:00.586656 5027 raft_consensus.cc:729] T a8328088928f4de19f7cf20189382d91 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.586869 5027 consensus_queue.cc:260] T a8328088928f4de19f7cf20189382d91 P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:00.587355 5027 ts_tablet_manager.cc:1398] T a8328088928f4de19f7cf20189382d91 P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:27:00.591459 5027 tablet_bootstrap.cc:492] T b67096e38f2e4c42a33e74f0145a04cc P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:27:00.596609 5027 tablet_bootstrap.cc:654] T b67096e38f2e4c42a33e74f0145a04cc P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:00.608623 5036 tablet_replica.cc:330] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db: stopping tablet replica
I20240502 21:27:00.609004 5036 raft_consensus.cc:2229] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:00.609457 5036 raft_consensus.cc:2258] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:00.610349 5036 ts_tablet_manager.cc:1875] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:00.610728 5026 tablet_bootstrap.cc:492] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:27:00.619594 5026 tablet_bootstrap.cc:654] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:00.623386 5035 tablet_replica.cc:330] T fe44f2c6df8a4597a7124ef400d5b3ee P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:27:00.623728 5035 raft_consensus.cc:2229] T fe44f2c6df8a4597a7124ef400d5b3ee P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:00.624032 5035 raft_consensus.cc:2258] T fe44f2c6df8a4597a7124ef400d5b3ee P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:00.625167 5035 ts_tablet_manager.cc:1875] T fe44f2c6df8a4597a7124ef400d5b3ee P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:00.632340 5034 tablet_bootstrap.cc:492] T b67096e38f2e4c42a33e74f0145a04cc P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:27:00.632797 31667 raft_consensus.cc:1203] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:00.634040 5034 tablet_bootstrap.cc:654] T b67096e38f2e4c42a33e74f0145a04cc P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:00.637075 5025 tablet_bootstrap.cc:492] T a8328088928f4de19f7cf20189382d91 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:27:00.639536 5025 tablet_bootstrap.cc:654] T a8328088928f4de19f7cf20189382d91 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:00.643584 5029 tablet_bootstrap.cc:492] T 18058b4a7c434ec8aff7832fad7ab7b7 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:27:00.643877 5029 ts_tablet_manager.cc:1367] T 18058b4a7c434ec8aff7832fad7ab7b7 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.065s user 0.000s sys 0.018s
I20240502 21:27:00.645020 5029 raft_consensus.cc:348] T 18058b4a7c434ec8aff7832fad7ab7b7 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:00.645413 5029 raft_consensus.cc:374] T 18058b4a7c434ec8aff7832fad7ab7b7 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:00.645562 5029 raft_consensus.cc:729] T 18058b4a7c434ec8aff7832fad7ab7b7 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.645934 5029 consensus_queue.cc:260] T 18058b4a7c434ec8aff7832fad7ab7b7 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:00.646402 5026 tablet_bootstrap.cc:492] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:27:00.646582 5026 ts_tablet_manager.cc:1367] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.037s user 0.007s sys 0.000s
I20240502 21:27:00.647310 5026 raft_consensus.cc:348] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:00.647567 5026 raft_consensus.cc:374] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:00.647697 5026 raft_consensus.cc:729] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.647826 5034 tablet_bootstrap.cc:492] T b67096e38f2e4c42a33e74f0145a04cc P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:27:00.648013 5034 ts_tablet_manager.cc:1367] T b67096e38f2e4c42a33e74f0145a04cc P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240502 21:27:00.647949 5026 consensus_queue.cc:260] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:00.650537 5025 tablet_bootstrap.cc:492] T a8328088928f4de19f7cf20189382d91 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:27:00.651211 5029 ts_tablet_manager.cc:1398] T 18058b4a7c434ec8aff7832fad7ab7b7 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 21:27:00.651486 4874 tablet_copy_client.cc:308] T dd0198178da640688230aeb5a1106cb4 P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.129:41039
I20240502 21:27:00.652269 5026 ts_tablet_manager.cc:1398] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 21:27:00.651194 5034 raft_consensus.cc:348] T b67096e38f2e4c42a33e74f0145a04cc P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:27:00.652065 31556 tablet_copy_service.cc:140] P a44f41b0b6644fb6aa29476aed8c964f: Received BeginTabletCopySession request for tablet dd0198178da640688230aeb5a1106cb4 from peer 1ade0354c44c4c37ae480c7a94b11e1b ({username='slave'} at 127.24.226.135:56009)
I20240502 21:27:00.652541 31556 tablet_copy_service.cc:161] P a44f41b0b6644fb6aa29476aed8c964f: Beginning new tablet copy session on tablet dd0198178da640688230aeb5a1106cb4 from peer 1ade0354c44c4c37ae480c7a94b11e1b at {username='slave'} at 127.24.226.135:56009: session id = 1ade0354c44c4c37ae480c7a94b11e1b-dd0198178da640688230aeb5a1106cb4
I20240502 21:27:00.652571 5034 raft_consensus.cc:374] T b67096e38f2e4c42a33e74f0145a04cc P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:00.652622 5034 raft_consensus.cc:729] T b67096e38f2e4c42a33e74f0145a04cc P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.654969 31556 tablet_copy_source_session.cc:205] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:00.655316 5034 consensus_queue.cc:260] T b67096e38f2e4c42a33e74f0145a04cc P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:27:00.655851 5034 ts_tablet_manager.cc:1398] T b67096e38f2e4c42a33e74f0145a04cc P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240502 21:27:00.652541 5025 ts_tablet_manager.cc:1367] T a8328088928f4de19f7cf20189382d91 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240502 21:27:00.659667 5025 raft_consensus.cc:348] T a8328088928f4de19f7cf20189382d91 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:00.659904 5025 raft_consensus.cc:374] T a8328088928f4de19f7cf20189382d91 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:00.660001 5025 raft_consensus.cc:729] T a8328088928f4de19f7cf20189382d91 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.660248 5025 consensus_queue.cc:260] T a8328088928f4de19f7cf20189382d91 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:00.660602 4874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd0198178da640688230aeb5a1106cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.660908 5025 ts_tablet_manager.cc:1398] T a8328088928f4de19f7cf20189382d91 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:00.668053 5026 tablet_bootstrap.cc:492] T b67096e38f2e4c42a33e74f0145a04cc P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:27:00.670657 5026 tablet_bootstrap.cc:654] T b67096e38f2e4c42a33e74f0145a04cc P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:00.673628 31386 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:00.684046 31775 tablet_service.cc:1467] Processing CreateTablet for tablet 5c302af336ed41aa8397f1b7479b6b7d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8a98fe5df3cf4207b7475fab13d8ed65]), partition=
I20240502 21:27:00.684229 31514 tablet_service.cc:1467] Processing CreateTablet for tablet 5c302af336ed41aa8397f1b7479b6b7d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8a98fe5df3cf4207b7475fab13d8ed65]), partition=
I20240502 21:27:00.684258 1814 tablet_service.cc:1467] Processing CreateTablet for tablet 5c302af336ed41aa8397f1b7479b6b7d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8a98fe5df3cf4207b7475fab13d8ed65]), partition=
I20240502 21:27:00.684439 31775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c302af336ed41aa8397f1b7479b6b7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.684619 31514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c302af336ed41aa8397f1b7479b6b7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.684725 1814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c302af336ed41aa8397f1b7479b6b7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.691033 5034 tablet_bootstrap.cc:492] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:27:00.692539 5034 tablet_bootstrap.cc:654] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:00.701772 5026 tablet_bootstrap.cc:492] T b67096e38f2e4c42a33e74f0145a04cc P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:27:00.701944 5026 ts_tablet_manager.cc:1367] T b67096e38f2e4c42a33e74f0145a04cc P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.003s
I20240502 21:27:00.703284 5027 tablet_bootstrap.cc:492] T b67096e38f2e4c42a33e74f0145a04cc P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:27:00.703457 5027 ts_tablet_manager.cc:1367] T b67096e38f2e4c42a33e74f0145a04cc P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.112s user 0.029s sys 0.051s
I20240502 21:27:00.703284 5026 raft_consensus.cc:348] T b67096e38f2e4c42a33e74f0145a04cc P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:27:00.704391 5036 ts_tablet_manager.cc:1888] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:00.706073 5050 ts_tablet_manager.cc:902] T ae56d18c629a4e5eb50911c89de4227c P d1406e92aec84ac585304e57bd37a40d: Initiating tablet copy from peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:27:00.708948 5025 tablet_bootstrap.cc:492] T 5c302af336ed41aa8397f1b7479b6b7d P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:27:00.709051 5035 ts_tablet_manager.cc:1888] T fe44f2c6df8a4597a7124ef400d5b3ee P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:00.709182 5035 log.cc:1196] T fe44f2c6df8a4597a7124ef400d5b3ee P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/fe44f2c6df8a4597a7124ef400d5b3ee
I20240502 21:27:00.709581 5035 ts_tablet_manager.cc:1909] T fe44f2c6df8a4597a7124ef400d5b3ee P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:27:00.710402 5025 tablet_bootstrap.cc:654] T 5c302af336ed41aa8397f1b7479b6b7d P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:00.710484 5036 log.cc:1196] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal/wals/fe44f2c6df8a4597a7124ef400d5b3ee
I20240502 21:27:00.711773 5036 ts_tablet_manager.cc:1909] T fe44f2c6df8a4597a7124ef400d5b3ee P 1e04157b98dd4fad98858c290b73c3db: Deleting consensus metadata
I20240502 21:27:00.711072 5027 raft_consensus.cc:348] T b67096e38f2e4c42a33e74f0145a04cc P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:27:00.711861 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet fe44f2c6df8a4597a7124ef400d5b3ee (table rebalancer_dynamic_table_new_name [id=b143f6f24b724c1cb9b64621ad271585]) successfully deleted
I20240502 21:27:00.712019 5027 raft_consensus.cc:374] T b67096e38f2e4c42a33e74f0145a04cc P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:00.712116 5027 raft_consensus.cc:729] T b67096e38f2e4c42a33e74f0145a04cc P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.704476 5026 raft_consensus.cc:374] T b67096e38f2e4c42a33e74f0145a04cc P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:00.712707 5026 raft_consensus.cc:729] T b67096e38f2e4c42a33e74f0145a04cc P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.712838 31381 catalog_manager.cc:4830] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): tablet fe44f2c6df8a4597a7124ef400d5b3ee (table rebalancer_dynamic_table_new_name [id=b143f6f24b724c1cb9b64621ad271585]) successfully deleted
I20240502 21:27:00.713140 5026 consensus_queue.cc:260] T b67096e38f2e4c42a33e74f0145a04cc P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:27:00.713892 5027 consensus_queue.cc:260] T b67096e38f2e4c42a33e74f0145a04cc P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:27:00.719693 5027 ts_tablet_manager.cc:1398] T b67096e38f2e4c42a33e74f0145a04cc P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20240502 21:27:00.719776 5026 ts_tablet_manager.cc:1398] T b67096e38f2e4c42a33e74f0145a04cc P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.017s user 0.000s sys 0.002s
I20240502 21:27:00.720074 5026 tablet_bootstrap.cc:492] T 5c302af336ed41aa8397f1b7479b6b7d P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:27:00.722890 5026 tablet_bootstrap.cc:654] T 5c302af336ed41aa8397f1b7479b6b7d P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:00.730891 5025 tablet_bootstrap.cc:492] T 5c302af336ed41aa8397f1b7479b6b7d P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:27:00.731060 5025 ts_tablet_manager.cc:1367] T 5c302af336ed41aa8397f1b7479b6b7d P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.000s
I20240502 21:27:00.731237 4874 tablet_copy_client.cc:783] T dd0198178da640688230aeb5a1106cb4 P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:00.731564 4874 tablet_copy_client.cc:647] T dd0198178da640688230aeb5a1106cb4 P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:00.731897 5025 raft_consensus.cc:348] T 5c302af336ed41aa8397f1b7479b6b7d P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:00.732170 5025 raft_consensus.cc:374] T 5c302af336ed41aa8397f1b7479b6b7d P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:00.732270 5025 raft_consensus.cc:729] T 5c302af336ed41aa8397f1b7479b6b7d P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.732574 5025 consensus_queue.cc:260] T 5c302af336ed41aa8397f1b7479b6b7d P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:00.733273 5025 ts_tablet_manager.cc:1398] T 5c302af336ed41aa8397f1b7479b6b7d P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:00.738373 4874 tablet_copy_client.cc:523] T dd0198178da640688230aeb5a1106cb4 P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:00.745251 4874 tablet_bootstrap.cc:492] T dd0198178da640688230aeb5a1106cb4 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:27:00.752862 5034 tablet_bootstrap.cc:492] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:27:00.753109 5034 ts_tablet_manager.cc:1367] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.062s user 0.006s sys 0.005s
I20240502 21:27:00.755316 5034 raft_consensus.cc:348] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:00.755616 5034 raft_consensus.cc:374] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:00.755685 5034 raft_consensus.cc:729] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.755892 5034 consensus_queue.cc:260] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:00.761175 5034 ts_tablet_manager.cc:1398] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 21:27:00.770272 4904 raft_consensus.cc:482] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:00.770471 4904 raft_consensus.cc:504] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:00.771296 4904 leader_election.cc:288] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:27:00.771942 1832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18058b4a7c434ec8aff7832fad7ab7b7" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:27:00.772163 1832 raft_consensus.cc:2454] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 0.
I20240502 21:27:00.772159 2230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18058b4a7c434ec8aff7832fad7ab7b7" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:27:00.772390 2230 raft_consensus.cc:2454] T 18058b4a7c434ec8aff7832fad7ab7b7 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 0.
I20240502 21:27:00.772822 2032 leader_election.cc:302] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 1e04157b98dd4fad98858c290b73c3db; no voters:
I20240502 21:27:00.778508 4986 raft_consensus.cc:482] T b67096e38f2e4c42a33e74f0145a04cc P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:00.778672 4986 raft_consensus.cc:504] T b67096e38f2e4c42a33e74f0145a04cc P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:27:00.779470 4986 leader_election.cc:288] T b67096e38f2e4c42a33e74f0145a04cc P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:27:00.780184 31796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b67096e38f2e4c42a33e74f0145a04cc" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:27:00.780416 31796 raft_consensus.cc:2454] T b67096e38f2e4c42a33e74f0145a04cc P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 0.
I20240502 21:27:00.780916 1769 leader_election.cc:302] T b67096e38f2e4c42a33e74f0145a04cc P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 1e04157b98dd4fad98858c290b73c3db; no voters:
I20240502 21:27:00.781365 4905 raft_consensus.cc:2781] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:00.781481 4905 raft_consensus.cc:482] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:00.781617 4905 raft_consensus.cc:3037] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:00.784255 4905 raft_consensus.cc:504] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:00.784904 4905 leader_election.cc:288] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 election: Requested vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:27:00.785620 1832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18058b4a7c434ec8aff7832fad7ab7b7" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:27:00.785847 1832 raft_consensus.cc:3037] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:00.786862 2230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18058b4a7c434ec8aff7832fad7ab7b7" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:27:00.787087 2230 raft_consensus.cc:3037] T 18058b4a7c434ec8aff7832fad7ab7b7 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:00.788538 1832 raft_consensus.cc:2454] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 1.
I20240502 21:27:00.789000 2032 leader_election.cc:302] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 1e04157b98dd4fad98858c290b73c3db; no voters:
I20240502 21:27:00.789322 4905 raft_consensus.cc:2781] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:00.789764 4905 raft_consensus.cc:686] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEADER]: Becoming Leader. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Running, Role: LEADER
I20240502 21:27:00.790127 4905 consensus_queue.cc:237] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:00.790410 4986 raft_consensus.cc:2781] T b67096e38f2e4c42a33e74f0145a04cc P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:00.790556 4986 raft_consensus.cc:482] T b67096e38f2e4c42a33e74f0145a04cc P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:00.790669 4986 raft_consensus.cc:3037] T b67096e38f2e4c42a33e74f0145a04cc P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:00.799743 1962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b67096e38f2e4c42a33e74f0145a04cc" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:27:00.800025 1962 raft_consensus.cc:2454] T b67096e38f2e4c42a33e74f0145a04cc P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 0.
I20240502 21:27:00.793890 4986 raft_consensus.cc:504] T b67096e38f2e4c42a33e74f0145a04cc P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:27:00.804113 31796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b67096e38f2e4c42a33e74f0145a04cc" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:27:00.804323 31796 raft_consensus.cc:3037] T b67096e38f2e4c42a33e74f0145a04cc P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:00.807588 31796 raft_consensus.cc:2454] T b67096e38f2e4c42a33e74f0145a04cc P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:27:00.808272 1769 leader_election.cc:302] T b67096e38f2e4c42a33e74f0145a04cc P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 1e04157b98dd4fad98858c290b73c3db; no voters:
I20240502 21:27:00.808722 4985 raft_consensus.cc:2781] T b67096e38f2e4c42a33e74f0145a04cc P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:00.808874 4985 raft_consensus.cc:686] T b67096e38f2e4c42a33e74f0145a04cc P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Becoming Leader. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Running, Role: LEADER
I20240502 21:27:00.809237 4985 consensus_queue.cc:237] T b67096e38f2e4c42a33e74f0145a04cc P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:27:00.815416 2230 raft_consensus.cc:2454] T 18058b4a7c434ec8aff7832fad7ab7b7 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 1.
I20240502 21:27:00.818104 31392 catalog_manager.cc:5484] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135). New cstate: current_term: 1 leader_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:00.819974 1964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b67096e38f2e4c42a33e74f0145a04cc" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf"
I20240502 21:27:00.820171 1964 raft_consensus.cc:3037] T b67096e38f2e4c42a33e74f0145a04cc P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:00.822284 4986 leader_election.cc:288] T b67096e38f2e4c42a33e74f0145a04cc P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Requested vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:27:00.822786 1964 raft_consensus.cc:2454] T b67096e38f2e4c42a33e74f0145a04cc P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:27:00.832741 5050 tablet_copy_client.cc:308] T ae56d18c629a4e5eb50911c89de4227c P d1406e92aec84ac585304e57bd37a40d: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.132:35933
I20240502 21:27:00.833815 31386 catalog_manager.cc:5484] T b67096e38f2e4c42a33e74f0145a04cc P 1e04157b98dd4fad98858c290b73c3db reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e04157b98dd4fad98858c290b73c3db (127.24.226.133). New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:00.835500 31947 tablet_copy_service.cc:140] P 2d50f33557944bb480ea32199d9ee4f1: Received BeginTabletCopySession request for tablet ae56d18c629a4e5eb50911c89de4227c from peer d1406e92aec84ac585304e57bd37a40d ({username='slave'} at 127.24.226.136:43115)
I20240502 21:27:00.835638 31947 tablet_copy_service.cc:161] P 2d50f33557944bb480ea32199d9ee4f1: Beginning new tablet copy session on tablet ae56d18c629a4e5eb50911c89de4227c from peer d1406e92aec84ac585304e57bd37a40d at {username='slave'} at 127.24.226.136:43115: session id = d1406e92aec84ac585304e57bd37a40d-ae56d18c629a4e5eb50911c89de4227c
I20240502 21:27:00.836589 4559 raft_consensus.cc:482] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:00.836704 4559 raft_consensus.cc:504] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:00.837229 4559 leader_election.cc:288] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:27:00.837745 1832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c302af336ed41aa8397f1b7479b6b7d" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:27:00.837872 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c302af336ed41aa8397f1b7479b6b7d" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:27:00.838001 31947 tablet_copy_source_session.cc:205] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:00.838057 31536 raft_consensus.cc:2454] T 5c302af336ed41aa8397f1b7479b6b7d P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
W20240502 21:27:00.839417 31732 leader_election.cc:341] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:00.839627 31733 leader_election.cc:302] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f; no voters: 1e04157b98dd4fad98858c290b73c3db
I20240502 21:27:00.839846 4559 raft_consensus.cc:2781] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:00.839917 4559 raft_consensus.cc:482] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:00.839967 4559 raft_consensus.cc:3037] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:00.841766 5050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae56d18c629a4e5eb50911c89de4227c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:00.844344 4559 raft_consensus.cc:504] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:00.845033 4559 leader_election.cc:288] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Requested vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:27:00.845461 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c302af336ed41aa8397f1b7479b6b7d" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:27:00.845672 31536 raft_consensus.cc:3037] T 5c302af336ed41aa8397f1b7479b6b7d P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:00.845746 1834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c302af336ed41aa8397f1b7479b6b7d" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
W20240502 21:27:00.846174 31732 leader_election.cc:341] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:00.848109 31536 raft_consensus.cc:2454] T 5c302af336ed41aa8397f1b7479b6b7d P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:27:00.850458 31733 leader_election.cc:302] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f; no voters: 1e04157b98dd4fad98858c290b73c3db
I20240502 21:27:00.850698 4559 raft_consensus.cc:2781] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:00.850817 4559 raft_consensus.cc:686] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Becoming Leader. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Running, Role: LEADER
I20240502 21:27:00.851006 4559 consensus_queue.cc:237] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:00.855047 5069 raft_consensus.cc:1050] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada: attempting to promote NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b to VOTER
I20240502 21:27:00.856030 5069 consensus_queue.cc:237] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } }
I20240502 21:27:00.858357 5026 tablet_bootstrap.cc:492] T 5c302af336ed41aa8397f1b7479b6b7d P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:27:00.858517 5026 ts_tablet_manager.cc:1367] T 5c302af336ed41aa8397f1b7479b6b7d P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.139s user 0.011s sys 0.034s
I20240502 21:27:00.859186 5050 tablet_copy_client.cc:783] T ae56d18c629a4e5eb50911c89de4227c P d1406e92aec84ac585304e57bd37a40d: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:00.859395 5050 tablet_copy_client.cc:647] T ae56d18c629a4e5eb50911c89de4227c P d1406e92aec84ac585304e57bd37a40d: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:00.860512 5026 raft_consensus.cc:348] T 5c302af336ed41aa8397f1b7479b6b7d P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:00.860798 5026 raft_consensus.cc:374] T 5c302af336ed41aa8397f1b7479b6b7d P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:00.860889 5026 raft_consensus.cc:729] T 5c302af336ed41aa8397f1b7479b6b7d P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:27:00.861075 5026 consensus_queue.cc:260] T 5c302af336ed41aa8397f1b7479b6b7d P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:00.861608 5026 ts_tablet_manager.cc:1398] T 5c302af336ed41aa8397f1b7479b6b7d P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:27:00.864491 5050 tablet_copy_client.cc:523] T ae56d18c629a4e5eb50911c89de4227c P d1406e92aec84ac585304e57bd37a40d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:00.867373 5071 ts_tablet_manager.cc:902] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c: Initiating tablet copy from peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:27:00.869913 2094 raft_consensus.cc:1261] T 94ba99a5798f4d5c9442590066896f93 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:00.870434 5073 consensus_queue.cc:1035] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:00.871619 5050 tablet_bootstrap.cc:492] T ae56d18c629a4e5eb50911c89de4227c P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:27:00.887854 31923 raft_consensus.cc:1203] T a699f448e116428f85cfee6ccbc98612 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:00.891525 31923 raft_consensus.cc:1261] T 94ba99a5798f4d5c9442590066896f93 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:00.892050 5073 consensus_queue.cc:1035] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:00.901398 31392 catalog_manager.cc:5484] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: term changed from 0 to 1, leader changed from <none> to 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131). New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:00.905582 31796 raft_consensus.cc:1261] T 94ba99a5798f4d5c9442590066896f93 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:00.906136 5065 consensus_queue.cc:1035] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:00.909116 5073 raft_consensus.cc:2932] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:27:00.915462 31796 raft_consensus.cc:2932] T 94ba99a5798f4d5c9442590066896f93 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:27:00.921343 31392 catalog_manager.cc:5484] T 94ba99a5798f4d5c9442590066896f93 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:27:00.921990 2093 raft_consensus.cc:2932] T 94ba99a5798f4d5c9442590066896f93 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:27:00.936462 4986 raft_consensus.cc:1050] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db: attempting to promote NON_VOTER 9dec645b473f40578e726f3e50e219bf to VOTER
I20240502 21:27:00.937218 4986 consensus_queue.cc:237] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:27:00.938141 31667 consensus_queue.cc:237] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } }
I20240502 21:27:00.939675 4985 raft_consensus.cc:482] T 5c302af336ed41aa8397f1b7479b6b7d P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:00.939857 4985 raft_consensus.cc:504] T 5c302af336ed41aa8397f1b7479b6b7d P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:00.940800 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c302af336ed41aa8397f1b7479b6b7d" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:27:00.941025 31536 raft_consensus.cc:2379] T 5c302af336ed41aa8397f1b7479b6b7d P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e04157b98dd4fad98858c290b73c3db in current term 1: Already voted for candidate 08d7a0680d244b55b93d17f9ab57b84c in this term.
I20240502 21:27:00.944542 4985 leader_election.cc:288] T 5c302af336ed41aa8397f1b7479b6b7d P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:27:00.944682 31797 raft_consensus.cc:1261] T 94ba99a5798f4d5c9442590066896f93 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:00.945425 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c302af336ed41aa8397f1b7479b6b7d" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:27:00.945674 2092 raft_consensus.cc:1261] T 94ba99a5798f4d5c9442590066896f93 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:00.946048 4968 consensus_queue.cc:1035] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:00.946388 4968 consensus_queue.cc:1035] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:00.947422 1769 leader_election.cc:302] T 5c302af336ed41aa8397f1b7479b6b7d P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db; no voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f
I20240502 21:27:00.947703 4744 raft_consensus.cc:3037] T 5c302af336ed41aa8397f1b7479b6b7d P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:00.950950 4744 raft_consensus.cc:2726] T 5c302af336ed41aa8397f1b7479b6b7d P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:27:00.955991 4498 raft_consensus.cc:482] T a8328088928f4de19f7cf20189382d91 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:00.956169 4498 raft_consensus.cc:504] T a8328088928f4de19f7cf20189382d91 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:00.957392 4498 leader_election.cc:288] T a8328088928f4de19f7cf20189382d91 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:27:00.957962 31533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8328088928f4de19f7cf20189382d91" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:27:00.958041 1965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8328088928f4de19f7cf20189382d91" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:27:00.958141 31533 raft_consensus.cc:2454] T a8328088928f4de19f7cf20189382d91 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:27:00.958218 1965 raft_consensus.cc:2454] T a8328088928f4de19f7cf20189382d91 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:27:00.958917 2164 leader_election.cc:302] T a8328088928f4de19f7cf20189382d91 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a44f41b0b6644fb6aa29476aed8c964f, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:27:00.959620 4498 raft_consensus.cc:2781] T a8328088928f4de19f7cf20189382d91 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:00.959722 4498 raft_consensus.cc:482] T a8328088928f4de19f7cf20189382d91 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:00.959810 4498 raft_consensus.cc:3037] T a8328088928f4de19f7cf20189382d91 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:00.962561 4498 raft_consensus.cc:504] T a8328088928f4de19f7cf20189382d91 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:00.963400 1962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8328088928f4de19f7cf20189382d91" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf"
I20240502 21:27:00.963630 1962 raft_consensus.cc:3037] T a8328088928f4de19f7cf20189382d91 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:00.965037 4968 raft_consensus.cc:2932] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:27:00.966498 1962 raft_consensus.cc:2454] T a8328088928f4de19f7cf20189382d91 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 1.
I20240502 21:27:00.969516 31382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 94ba99a5798f4d5c9442590066896f93 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:00.970484 2161 leader_election.cc:302] T a8328088928f4de19f7cf20189382d91 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dec645b473f40578e726f3e50e219bf, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:27:00.972014 4966 raft_consensus.cc:2781] T a8328088928f4de19f7cf20189382d91 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:00.972153 4966 raft_consensus.cc:686] T a8328088928f4de19f7cf20189382d91 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Becoming Leader. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Running, Role: LEADER
I20240502 21:27:00.972460 4966 consensus_queue.cc:237] T a8328088928f4de19f7cf20189382d91 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:00.975935 31386 catalog_manager.cc:5484] T 94ba99a5798f4d5c9442590066896f93 P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: config changed from index 5 to 6, VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) evicted. New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:00.979760 4498 leader_election.cc:288] T a8328088928f4de19f7cf20189382d91 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Requested vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:27:00.980278 31797 raft_consensus.cc:2932] T 94ba99a5798f4d5c9442590066896f93 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:27:00.984223 31906 tablet_service.cc:1514] Processing DeleteTablet for tablet 94ba99a5798f4d5c9442590066896f93 with delete_type TABLET_DATA_TOMBSTONED (TS 2d50f33557944bb480ea32199d9ee4f1 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47824
I20240502 21:27:00.988359 31392 catalog_manager.cc:5484] T a8328088928f4de19f7cf20189382d91 P d1406e92aec84ac585304e57bd37a40d reported cstate change: term changed from 0 to 1, leader changed from <none> to d1406e92aec84ac585304e57bd37a40d (127.24.226.136). New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:00.992982 2092 raft_consensus.cc:1261] T 3e6657723fe84e749dea768de44638e7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:00.993116 1962 raft_consensus.cc:1261] T 3e6657723fe84e749dea768de44638e7 P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:00.993687 31533 raft_consensus.cc:1261] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:00.993806 4985 consensus_queue.cc:1035] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:00.994522 4985 consensus_queue.cc:1035] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:00.995963 4985 consensus_queue.cc:1035] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:00.999447 2093 raft_consensus.cc:2932] T 94ba99a5798f4d5c9442590066896f93 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:27:01.005692 5085 ts_tablet_manager.cc:902] T 59f36d947f4341f39b6969cc1e065b6c P 9dec645b473f40578e726f3e50e219bf: Initiating tablet copy from peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:27:01.006065 5086 ts_tablet_manager.cc:902] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f: Initiating tablet copy from peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:27:01.008407 4986 raft_consensus.cc:2932] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:27:01.013968 4874 tablet_bootstrap.cc:492] T dd0198178da640688230aeb5a1106cb4 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:01.014364 4874 tablet_bootstrap.cc:492] T dd0198178da640688230aeb5a1106cb4 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap complete.
I20240502 21:27:01.014662 4874 ts_tablet_manager.cc:1367] T dd0198178da640688230aeb5a1106cb4 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.270s user 0.038s sys 0.006s
I20240502 21:27:01.015419 4874 raft_consensus.cc:348] T dd0198178da640688230aeb5a1106cb4 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } }
I20240502 21:27:01.015681 4874 raft_consensus.cc:729] T dd0198178da640688230aeb5a1106cb4 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: LEARNER
I20240502 21:27:01.013192 2092 raft_consensus.cc:2932] T 3e6657723fe84e749dea768de44638e7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:27:01.015964 4874 consensus_queue.cc:260] T dd0198178da640688230aeb5a1106cb4 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } }
I20240502 21:27:01.016651 4874 ts_tablet_manager.cc:1398] T dd0198178da640688230aeb5a1106cb4 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 21:27:01.017707 31556 tablet_copy_service.cc:342] P a44f41b0b6644fb6aa29476aed8c964f: Request end of tablet copy session 1ade0354c44c4c37ae480c7a94b11e1b-dd0198178da640688230aeb5a1106cb4 received from {username='slave'} at 127.24.226.135:56009
I20240502 21:27:01.017800 31556 tablet_copy_service.cc:433] P a44f41b0b6644fb6aa29476aed8c964f: ending tablet copy session 1ade0354c44c4c37ae480c7a94b11e1b-dd0198178da640688230aeb5a1106cb4 on tablet dd0198178da640688230aeb5a1106cb4 with peer 1ade0354c44c4c37ae480c7a94b11e1b
I20240502 21:27:01.021056 5071 tablet_copy_client.cc:308] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.133:39863
I20240502 21:27:01.024809 1854 tablet_copy_service.cc:140] P 1e04157b98dd4fad98858c290b73c3db: Received BeginTabletCopySession request for tablet 9bc57b8a0388416290997df544e5cc0b from peer 08d7a0680d244b55b93d17f9ab57b84c ({username='slave'} at 127.24.226.131:42585)
I20240502 21:27:01.024997 1854 tablet_copy_service.cc:161] P 1e04157b98dd4fad98858c290b73c3db: Beginning new tablet copy session on tablet 9bc57b8a0388416290997df544e5cc0b from peer 08d7a0680d244b55b93d17f9ab57b84c at {username='slave'} at 127.24.226.131:42585: session id = 08d7a0680d244b55b93d17f9ab57b84c-9bc57b8a0388416290997df544e5cc0b
I20240502 21:27:01.026693 31386 catalog_manager.cc:5484] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db reported cstate change: config changed from index 2 to 3, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:01.028198 1854 tablet_copy_source_session.cc:205] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:01.031026 5092 tablet_replica.cc:330] T 94ba99a5798f4d5c9442590066896f93 P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:27:01.031677 5092 raft_consensus.cc:2229] T 94ba99a5798f4d5c9442590066896f93 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:01.031476 31536 raft_consensus.cc:2932] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:27:01.031917 5092 pending_rounds.cc:62] T 94ba99a5798f4d5c9442590066896f93 P 2d50f33557944bb480ea32199d9ee4f1: Trying to abort 1 pending ops.
I20240502 21:27:01.032145 5092 pending_rounds.cc:68] T 94ba99a5798f4d5c9442590066896f93 P 2d50f33557944bb480ea32199d9ee4f1: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023350664624250880 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "94ba99a5798f4d5c9442590066896f93" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } } }
I20240502 21:27:01.032620 5092 raft_consensus.cc:2894] T 94ba99a5798f4d5c9442590066896f93 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 21:27:01.032773 5092 raft_consensus.cc:2866] T 94ba99a5798f4d5c9442590066896f93 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 21:27:01.033308 5071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bc57b8a0388416290997df544e5cc0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.037269 31533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8328088928f4de19f7cf20189382d91" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:27:01.037460 31533 raft_consensus.cc:3037] T a8328088928f4de19f7cf20189382d91 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.039849 31533 raft_consensus.cc:2454] T a8328088928f4de19f7cf20189382d91 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 1.
I20240502 21:27:01.046898 1962 raft_consensus.cc:2932] T 3e6657723fe84e749dea768de44638e7 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:27:01.040760 31392 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:01.052131 5086 tablet_copy_client.cc:308] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.134:43753
I20240502 21:27:01.058992 5092 raft_consensus.cc:2258] T 94ba99a5798f4d5c9442590066896f93 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:01.063001 5071 tablet_copy_client.cc:783] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:01.063793 5092 ts_tablet_manager.cc:1875] T 94ba99a5798f4d5c9442590066896f93 P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:01.069064 5085 tablet_copy_client.cc:308] T 59f36d947f4341f39b6969cc1e065b6c P 9dec645b473f40578e726f3e50e219bf: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.136:34907
I20240502 21:27:01.069763 2251 tablet_copy_service.cc:140] P d1406e92aec84ac585304e57bd37a40d: Received BeginTabletCopySession request for tablet 59f36d947f4341f39b6969cc1e065b6c from peer 9dec645b473f40578e726f3e50e219bf ({username='slave'} at 127.24.226.134:57305)
I20240502 21:27:01.069972 2251 tablet_copy_service.cc:161] P d1406e92aec84ac585304e57bd37a40d: Beginning new tablet copy session on tablet 59f36d947f4341f39b6969cc1e065b6c from peer 9dec645b473f40578e726f3e50e219bf at {username='slave'} at 127.24.226.134:57305: session id = 9dec645b473f40578e726f3e50e219bf-59f36d947f4341f39b6969cc1e065b6c
I20240502 21:27:01.072635 2251 tablet_copy_source_session.cc:205] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:01.074870 1986 tablet_copy_service.cc:140] P 9dec645b473f40578e726f3e50e219bf: Received BeginTabletCopySession request for tablet 51551a1967b74094b373cc030e2bf850 from peer a44f41b0b6644fb6aa29476aed8c964f ({username='slave'} at 127.24.226.129:41327)
I20240502 21:27:01.075088 1986 tablet_copy_service.cc:161] P 9dec645b473f40578e726f3e50e219bf: Beginning new tablet copy session on tablet 51551a1967b74094b373cc030e2bf850 from peer a44f41b0b6644fb6aa29476aed8c964f at {username='slave'} at 127.24.226.129:41327: session id = a44f41b0b6644fb6aa29476aed8c964f-51551a1967b74094b373cc030e2bf850
I20240502 21:27:01.075562 5085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59f36d947f4341f39b6969cc1e065b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.077844 1986 tablet_copy_source_session.cc:205] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:01.080534 5086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51551a1967b74094b373cc030e2bf850. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.082123 5071 tablet_copy_client.cc:647] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:01.082906 31536 raft_consensus.cc:1261] T 5c302af336ed41aa8397f1b7479b6b7d P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:01.083513 4559 consensus_queue.cc:1035] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:01.085996 5071 tablet_copy_client.cc:523] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:01.087455 5086 tablet_copy_client.cc:783] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:01.087627 5086 tablet_copy_client.cc:647] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:01.088860 1832 raft_consensus.cc:1261] T 5c302af336ed41aa8397f1b7479b6b7d P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:01.089515 4559 consensus_queue.cc:1035] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:01.090287 5085 tablet_copy_client.cc:783] T 59f36d947f4341f39b6969cc1e065b6c P 9dec645b473f40578e726f3e50e219bf: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:01.090626 5085 tablet_copy_client.cc:647] T 59f36d947f4341f39b6969cc1e065b6c P 9dec645b473f40578e726f3e50e219bf: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:01.091972 5092 ts_tablet_manager.cc:1888] T 94ba99a5798f4d5c9442590066896f93 P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:01.097164 5086 tablet_copy_client.cc:523] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:01.097652 5085 tablet_copy_client.cc:523] T 59f36d947f4341f39b6969cc1e065b6c P 9dec645b473f40578e726f3e50e219bf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:01.098042 5071 tablet_bootstrap.cc:492] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:27:01.099654 31392 catalog_manager.cc:5576] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863) does not have the latest schema for tablet 5c302af336ed41aa8397f1b7479b6b7d (table rebalancer_dynamic_table [id=8a98fe5df3cf4207b7475fab13d8ed65]). Expected version 1 got 0
I20240502 21:27:01.100330 5086 tablet_bootstrap.cc:492] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:27:01.101100 5092 log.cc:1196] T 94ba99a5798f4d5c9442590066896f93 P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/94ba99a5798f4d5c9442590066896f93
I20240502 21:27:01.102869 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet 94ba99a5798f4d5c9442590066896f93 (table rebalance_test_table_5 [id=9e294586148643e080e91357bcbffb17]) successfully deleted
I20240502 21:27:01.107164 5115 tablet.cc:1722] T 5c302af336ed41aa8397f1b7479b6b7d P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:01.107702 5115 tablet.cc:1620] T 5c302af336ed41aa8397f1b7479b6b7d P 1e04157b98dd4fad98858c290b73c3db: MemRowSet was empty: no flush needed.
I20240502 21:27:01.113245 5085 tablet_bootstrap.cc:492] T 59f36d947f4341f39b6969cc1e065b6c P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:27:01.116976 31392 catalog_manager.cc:5576] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863) does not have the latest schema for tablet 5c302af336ed41aa8397f1b7479b6b7d (table rebalancer_dynamic_table [id=8a98fe5df3cf4207b7475fab13d8ed65]). Expected version 1 got 0
I20240502 21:27:01.123067 5086 tablet_bootstrap.cc:492] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:01.129218 5114 tablet.cc:1722] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:01.131803 5114 tablet.cc:1620] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c: MemRowSet was empty: no flush needed.
I20240502 21:27:01.132589 31386 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=8a98fe5df3cf4207b7475fab13d8ed65] alter complete (version 1)
I20240502 21:27:01.132652 5071 tablet_bootstrap.cc:492] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:01.123427 5086 tablet_bootstrap.cc:492] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap complete.
I20240502 21:27:01.133263 5086 ts_tablet_manager.cc:1367] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.004s
I20240502 21:27:01.133502 5071 tablet_bootstrap.cc:492] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap complete.
I20240502 21:27:01.133775 5071 ts_tablet_manager.cc:1367] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.004s
I20240502 21:27:01.134024 5086 raft_consensus.cc:348] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } }
I20240502 21:27:01.134236 5086 raft_consensus.cc:729] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: LEARNER
I20240502 21:27:01.134434 5086 consensus_queue.cc:260] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: NON_VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: true } }
I20240502 21:27:01.134594 31392 catalog_manager.cc:5576] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807) does not have the latest schema for tablet 5c302af336ed41aa8397f1b7479b6b7d (table rebalancer_dynamic_table [id=8a98fe5df3cf4207b7475fab13d8ed65]). Expected version 1 got 0
I20240502 21:27:01.134754 5071 raft_consensus.cc:348] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } }
I20240502 21:27:01.135145 5071 raft_consensus.cc:729] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: LEARNER
I20240502 21:27:01.135452 5071 consensus_queue.cc:260] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } }
I20240502 21:27:01.136277 5071 ts_tablet_manager.cc:1398] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:01.137071 5086 ts_tablet_manager.cc:1398] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:01.141746 1986 tablet_copy_service.cc:342] P 9dec645b473f40578e726f3e50e219bf: Request end of tablet copy session a44f41b0b6644fb6aa29476aed8c964f-51551a1967b74094b373cc030e2bf850 received from {username='slave'} at 127.24.226.129:41327
I20240502 21:27:01.141867 1986 tablet_copy_service.cc:433] P 9dec645b473f40578e726f3e50e219bf: ending tablet copy session a44f41b0b6644fb6aa29476aed8c964f-51551a1967b74094b373cc030e2bf850 on tablet 51551a1967b74094b373cc030e2bf850 with peer a44f41b0b6644fb6aa29476aed8c964f
I20240502 21:27:01.143266 31392 catalog_manager.cc:5576] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039) does not have the latest schema for tablet 5c302af336ed41aa8397f1b7479b6b7d (table rebalancer_dynamic_table [id=8a98fe5df3cf4207b7475fab13d8ed65]). Expected version 1 got 0
I20240502 21:27:01.148146 5085 tablet_bootstrap.cc:492] T 59f36d947f4341f39b6969cc1e065b6c P 9dec645b473f40578e726f3e50e219bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:01.148605 5085 tablet_bootstrap.cc:492] T 59f36d947f4341f39b6969cc1e065b6c P 9dec645b473f40578e726f3e50e219bf: Bootstrap complete.
I20240502 21:27:01.148866 5085 ts_tablet_manager.cc:1367] T 59f36d947f4341f39b6969cc1e065b6c P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.004s
I20240502 21:27:01.150259 5085 raft_consensus.cc:348] T 59f36d947f4341f39b6969cc1e065b6c P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:27:01.151871 5118 ts_tablet_manager.cc:902] T f74ea10ab8a2459794f816707b87f494 P 08d7a0680d244b55b93d17f9ab57b84c: Initiating tablet copy from peer 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:27:01.152359 5085 raft_consensus.cc:729] T 59f36d947f4341f39b6969cc1e065b6c P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: LEARNER
I20240502 21:27:01.152838 5085 consensus_queue.cc:260] T 59f36d947f4341f39b6969cc1e065b6c P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:27:01.154255 1854 tablet_copy_service.cc:342] P 1e04157b98dd4fad98858c290b73c3db: Request end of tablet copy session 08d7a0680d244b55b93d17f9ab57b84c-9bc57b8a0388416290997df544e5cc0b received from {username='slave'} at 127.24.226.131:42585
I20240502 21:27:01.154362 1854 tablet_copy_service.cc:433] P 1e04157b98dd4fad98858c290b73c3db: ending tablet copy session 08d7a0680d244b55b93d17f9ab57b84c-9bc57b8a0388416290997df544e5cc0b on tablet 9bc57b8a0388416290997df544e5cc0b with peer 08d7a0680d244b55b93d17f9ab57b84c
I20240502 21:27:01.155706 5085 ts_tablet_manager.cc:1398] T 59f36d947f4341f39b6969cc1e065b6c P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 21:27:01.156570 2251 tablet_copy_service.cc:342] P d1406e92aec84ac585304e57bd37a40d: Request end of tablet copy session 9dec645b473f40578e726f3e50e219bf-59f36d947f4341f39b6969cc1e065b6c received from {username='slave'} at 127.24.226.134:57305
I20240502 21:27:01.156684 2251 tablet_copy_service.cc:433] P d1406e92aec84ac585304e57bd37a40d: ending tablet copy session 9dec645b473f40578e726f3e50e219bf-59f36d947f4341f39b6969cc1e065b6c on tablet 59f36d947f4341f39b6969cc1e065b6c with peer 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:01.162686 3818 raft_consensus.cc:1050] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf: attempting to promote NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada to VOTER
I20240502 21:27:01.163470 3818 consensus_queue.cc:237] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } }
I20240502 21:27:01.182541 5123 tablet.cc:1722] T 5c302af336ed41aa8397f1b7479b6b7d P a44f41b0b6644fb6aa29476aed8c964f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:01.183601 31665 raft_consensus.cc:1261] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:01.183091 5123 tablet.cc:1620] T 5c302af336ed41aa8397f1b7479b6b7d P a44f41b0b6644fb6aa29476aed8c964f: MemRowSet was empty: no flush needed.
I20240502 21:27:01.184379 4801 consensus_queue.cc:1035] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:01.194022 2230 raft_consensus.cc:1261] T b20c2bb60f6149458a62ee443b8cf3ab P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:01.195027 4801 consensus_queue.cc:1035] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:01.198573 5118 tablet_copy_client.cc:308] T f74ea10ab8a2459794f816707b87f494 P 08d7a0680d244b55b93d17f9ab57b84c: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.135:38569
I20240502 21:27:01.199234 2118 tablet_copy_service.cc:140] P 1ade0354c44c4c37ae480c7a94b11e1b: Received BeginTabletCopySession request for tablet f74ea10ab8a2459794f816707b87f494 from peer 08d7a0680d244b55b93d17f9ab57b84c ({username='slave'} at 127.24.226.131:60059)
I20240502 21:27:01.199402 2118 tablet_copy_service.cc:161] P 1ade0354c44c4c37ae480c7a94b11e1b: Beginning new tablet copy session on tablet f74ea10ab8a2459794f816707b87f494 from peer 08d7a0680d244b55b93d17f9ab57b84c at {username='slave'} at 127.24.226.131:60059: session id = 08d7a0680d244b55b93d17f9ab57b84c-f74ea10ab8a2459794f816707b87f494
I20240502 21:27:01.199395 31392 catalog_manager.cc:5576] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039) does not have the latest schema for tablet 5c302af336ed41aa8397f1b7479b6b7d (table rebalancer_dynamic_table [id=8a98fe5df3cf4207b7475fab13d8ed65]). Expected version 1 got 0
I20240502 21:27:01.202097 2118 tablet_copy_source_session.cc:205] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:01.203394 5118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f74ea10ab8a2459794f816707b87f494. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.210454 5118 tablet_copy_client.cc:783] T f74ea10ab8a2459794f816707b87f494 P 08d7a0680d244b55b93d17f9ab57b84c: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:01.210680 5118 tablet_copy_client.cc:647] T f74ea10ab8a2459794f816707b87f494 P 08d7a0680d244b55b93d17f9ab57b84c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:01.212280 5118 tablet_copy_client.cc:523] T f74ea10ab8a2459794f816707b87f494 P 08d7a0680d244b55b93d17f9ab57b84c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:01.213793 31797 raft_consensus.cc:1203] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:01.215687 5118 tablet_bootstrap.cc:492] T f74ea10ab8a2459794f816707b87f494 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:27:01.218997 4905 consensus_queue.cc:1035] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:01.225337 5050 tablet_bootstrap.cc:492] T ae56d18c629a4e5eb50911c89de4227c P d1406e92aec84ac585304e57bd37a40d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:01.225855 5050 tablet_bootstrap.cc:492] T ae56d18c629a4e5eb50911c89de4227c P d1406e92aec84ac585304e57bd37a40d: Bootstrap complete.
I20240502 21:27:01.226141 5050 ts_tablet_manager.cc:1367] T ae56d18c629a4e5eb50911c89de4227c P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.355s user 0.059s sys 0.029s
I20240502 21:27:01.233482 31533 raft_consensus.cc:1261] T b20c2bb60f6149458a62ee443b8cf3ab P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:01.234673 5125 consensus_queue.cc:1035] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:01.236544 31392 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:01.247743 5118 tablet_bootstrap.cc:492] T f74ea10ab8a2459794f816707b87f494 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:01.248071 5118 tablet_bootstrap.cc:492] T f74ea10ab8a2459794f816707b87f494 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap complete.
I20240502 21:27:01.248275 5118 ts_tablet_manager.cc:1367] T f74ea10ab8a2459794f816707b87f494 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.003s
I20240502 21:27:01.249020 5114 tablet.cc:1722] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:01.249434 5114 tablet.cc:1620] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c: MemRowSet was empty: no flush needed.
I20240502 21:27:01.251057 5123 tablet.cc:1722] T 5c302af336ed41aa8397f1b7479b6b7d P a44f41b0b6644fb6aa29476aed8c964f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:01.251534 5123 tablet.cc:1620] T 5c302af336ed41aa8397f1b7479b6b7d P a44f41b0b6644fb6aa29476aed8c964f: MemRowSet was empty: no flush needed.
I20240502 21:27:01.254027 5050 raft_consensus.cc:348] T ae56d18c629a4e5eb50911c89de4227c P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } }
I20240502 21:27:01.258260 5118 raft_consensus.cc:348] T f74ea10ab8a2459794f816707b87f494 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } }
I20240502 21:27:01.258457 5118 raft_consensus.cc:729] T f74ea10ab8a2459794f816707b87f494 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: LEARNER
I20240502 21:27:01.258622 5118 consensus_queue.cc:260] T f74ea10ab8a2459794f816707b87f494 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } }
I20240502 21:27:01.259126 5118 ts_tablet_manager.cc:1398] T f74ea10ab8a2459794f816707b87f494 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240502 21:27:01.259224 5050 raft_consensus.cc:729] T ae56d18c629a4e5eb50911c89de4227c P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: LEARNER
I20240502 21:27:01.259475 5050 consensus_queue.cc:260] T ae56d18c629a4e5eb50911c89de4227c P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: NON_VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: true } }
I20240502 21:27:01.259773 5115 tablet.cc:1722] T 5c302af336ed41aa8397f1b7479b6b7d P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:01.260303 5115 tablet.cc:1620] T 5c302af336ed41aa8397f1b7479b6b7d P 1e04157b98dd4fad98858c290b73c3db: MemRowSet was empty: no flush needed.
I20240502 21:27:01.260813 5050 ts_tablet_manager.cc:1398] T ae56d18c629a4e5eb50911c89de4227c P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.034s user 0.003s sys 0.000s
I20240502 21:27:01.263000 31947 tablet_copy_service.cc:342] P 2d50f33557944bb480ea32199d9ee4f1: Request end of tablet copy session d1406e92aec84ac585304e57bd37a40d-ae56d18c629a4e5eb50911c89de4227c received from {username='slave'} at 127.24.226.136:43115
I20240502 21:27:01.263134 31947 tablet_copy_service.cc:433] P 2d50f33557944bb480ea32199d9ee4f1: ending tablet copy session d1406e92aec84ac585304e57bd37a40d-ae56d18c629a4e5eb50911c89de4227c on tablet ae56d18c629a4e5eb50911c89de4227c with peer d1406e92aec84ac585304e57bd37a40d
I20240502 21:27:01.269482 2118 tablet_copy_service.cc:342] P 1ade0354c44c4c37ae480c7a94b11e1b: Request end of tablet copy session 08d7a0680d244b55b93d17f9ab57b84c-f74ea10ab8a2459794f816707b87f494 received from {username='slave'} at 127.24.226.131:60059
I20240502 21:27:01.269614 2118 tablet_copy_service.cc:433] P 1ade0354c44c4c37ae480c7a94b11e1b: ending tablet copy session 08d7a0680d244b55b93d17f9ab57b84c-f74ea10ab8a2459794f816707b87f494 on tablet f74ea10ab8a2459794f816707b87f494 with peer 08d7a0680d244b55b93d17f9ab57b84c
I20240502 21:27:01.270440 31386 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=8a98fe5df3cf4207b7475fab13d8ed65] alter complete (version 2)
I20240502 21:27:01.275593 4905 consensus_queue.cc:1035] T 18058b4a7c434ec8aff7832fad7ab7b7 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:01.277175 5134 raft_consensus.cc:2932] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:27:01.291713 2230 raft_consensus.cc:2932] T b20c2bb60f6149458a62ee443b8cf3ab P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:27:01.292524 4986 consensus_queue.cc:1035] T b67096e38f2e4c42a33e74f0145a04cc P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:01.300168 31665 raft_consensus.cc:2932] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:27:01.312090 31392 catalog_manager.cc:5484] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9dec645b473f40578e726f3e50e219bf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:27:01.317644 31533 raft_consensus.cc:2932] T b20c2bb60f6149458a62ee443b8cf3ab P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:27:01.324813 4985 consensus_queue.cc:1035] T b67096e38f2e4c42a33e74f0145a04cc P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:01.343974 4966 raft_consensus.cc:1050] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d: attempting to promote NON_VOTER 2d50f33557944bb480ea32199d9ee4f1 to VOTER
I20240502 21:27:01.344852 4966 consensus_queue.cc:237] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } }
I20240502 21:27:01.355711 1962 consensus_queue.cc:237] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } }
I20240502 21:27:01.357961 31665 raft_consensus.cc:1261] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:01.358779 5134 consensus_queue.cc:1035] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:01.360481 31533 raft_consensus.cc:1261] T b20c2bb60f6149458a62ee443b8cf3ab P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:01.379999 5134 consensus_queue.cc:1035] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:01.381086 3818 raft_consensus.cc:2932] T b20c2bb60f6149458a62ee443b8cf3ab P 9dec645b473f40578e726f3e50e219bf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:27:01.383492 31665 raft_consensus.cc:2932] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:27:01.385076 31389 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:01.388541 31533 raft_consensus.cc:2932] T b20c2bb60f6149458a62ee443b8cf3ab P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:27:01.389439 31392 catalog_manager.cc:5484] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: config changed from index 3 to 4, VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) evicted. New cstate: current_term: 1 leader_uuid: "9dec645b473f40578e726f3e50e219bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:27:01.397998 5114 tablet.cc:1722] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:01.399117 5115 tablet.cc:1722] T 5c302af336ed41aa8397f1b7479b6b7d P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:01.402891 31379 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b20c2bb60f6149458a62ee443b8cf3ab with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:01.405301 5156 consensus_queue.cc:1035] T a8328088928f4de19f7cf20189382d91 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:01.407817 31392 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table_new_name [id=8a98fe5df3cf4207b7475fab13d8ed65] alter complete (version 3)
I20240502 21:27:01.411296 5123 tablet.cc:1722] T 5c302af336ed41aa8397f1b7479b6b7d P a44f41b0b6644fb6aa29476aed8c964f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:01.414652 31923 raft_consensus.cc:1261] T a699f448e116428f85cfee6ccbc98612 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEARNER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:01.416463 5154 consensus_queue.cc:1035] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:01.418155 31797 raft_consensus.cc:1261] T a699f448e116428f85cfee6ccbc98612 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20240502 21:27:01.426563 2161 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.226.134:43753, user_credentials={real_user=slave}} blocked reactor thread for 66094.6us
I20240502 21:27:01.427398 5151 consensus_queue.cc:1035] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20240502 21:27:01.443389 2208 tablet_service.cc:1514] Processing DeleteTablet for tablet b20c2bb60f6149458a62ee443b8cf3ab with delete_type TABLET_DATA_TOMBSTONED (TS d1406e92aec84ac585304e57bd37a40d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36984
I20240502 21:27:01.443850 1962 raft_consensus.cc:1203] T 59f36d947f4341f39b6969cc1e065b6c P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:01.450721 2092 raft_consensus.cc:1203] T dd0198178da640688230aeb5a1106cb4 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:01.463418 5162 tablet_replica.cc:330] T b20c2bb60f6149458a62ee443b8cf3ab P d1406e92aec84ac585304e57bd37a40d: stopping tablet replica
I20240502 21:27:01.463980 5162 raft_consensus.cc:2229] T b20c2bb60f6149458a62ee443b8cf3ab P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:01.464372 5162 raft_consensus.cc:2258] T b20c2bb60f6149458a62ee443b8cf3ab P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:01.465406 5162 ts_tablet_manager.cc:1875] T b20c2bb60f6149458a62ee443b8cf3ab P d1406e92aec84ac585304e57bd37a40d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:01.466414 5154 consensus_queue.cc:1035] T a8328088928f4de19f7cf20189382d91 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:01.473129 31536 raft_consensus.cc:1261] T a699f448e116428f85cfee6ccbc98612 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:01.475754 5158 consensus_queue.cc:1035] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:01.479171 5154 raft_consensus.cc:2932] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:27:01.483669 31923 raft_consensus.cc:2932] T a699f448e116428f85cfee6ccbc98612 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:27:01.486552 31393 catalog_manager.cc:5484] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d reported cstate change: config changed from index 2 to 3, 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:01.493546 31797 raft_consensus.cc:2932] T a699f448e116428f85cfee6ccbc98612 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:27:01.506527 31536 raft_consensus.cc:2932] T a699f448e116428f85cfee6ccbc98612 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:27:01.510954 31392 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:01.511238 31392 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:01.515688 2224 consensus_queue.cc:237] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } }
I20240502 21:27:01.517761 31923 raft_consensus.cc:1261] T a699f448e116428f85cfee6ccbc98612 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:01.518296 5153 consensus_queue.cc:1035] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:01.518436 5162 ts_tablet_manager.cc:1888] T b20c2bb60f6149458a62ee443b8cf3ab P d1406e92aec84ac585304e57bd37a40d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:27:01.519570 5162 log.cc:1196] T b20c2bb60f6149458a62ee443b8cf3ab P d1406e92aec84ac585304e57bd37a40d: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal/wals/b20c2bb60f6149458a62ee443b8cf3ab
I20240502 21:27:01.521011 31381 catalog_manager.cc:4830] TS d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): tablet b20c2bb60f6149458a62ee443b8cf3ab (table rebalancer_extra_table_10 [id=112fc5f7af4e4c8080926e0a6128b468]) successfully deleted
I20240502 21:27:01.522321 31392 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 5c302af336ed41aa8397f1b7479b6b7d
I20240502 21:27:01.522351 5153 raft_consensus.cc:2932] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:27:01.525362 1814 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c302af336ed41aa8397f1b7479b6b7d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:01 UTC) from {username='slave'} at 127.0.0.1:47510
I20240502 21:27:01.526034 31775 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c302af336ed41aa8397f1b7479b6b7d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:01 UTC) from {username='slave'} at 127.0.0.1:51596
I20240502 21:27:01.529999 31533 raft_consensus.cc:1203] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:01.532760 31514 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c302af336ed41aa8397f1b7479b6b7d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:01 UTC) from {username='slave'} at 127.0.0.1:45194
I20240502 21:27:01.540797 5172 tablet_replica.cc:330] T 5c302af336ed41aa8397f1b7479b6b7d P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:27:01.541033 5172 raft_consensus.cc:2229] T 5c302af336ed41aa8397f1b7479b6b7d P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:01.541424 5172 raft_consensus.cc:2258] T 5c302af336ed41aa8397f1b7479b6b7d P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:01.541296 31923 raft_consensus.cc:2932] T a699f448e116428f85cfee6ccbc98612 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:27:01.542443 5172 ts_tablet_manager.cc:1875] T 5c302af336ed41aa8397f1b7479b6b7d P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:01.548259 5172 ts_tablet_manager.cc:1888] T 5c302af336ed41aa8397f1b7479b6b7d P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:01.548458 5172 log.cc:1196] T 5c302af336ed41aa8397f1b7479b6b7d P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/5c302af336ed41aa8397f1b7479b6b7d
I20240502 21:27:01.548810 5172 ts_tablet_manager.cc:1909] T 5c302af336ed41aa8397f1b7479b6b7d P a44f41b0b6644fb6aa29476aed8c964f: Deleting consensus metadata
I20240502 21:27:01.551877 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet 5c302af336ed41aa8397f1b7479b6b7d (table rebalancer_dynamic_table_new_name [id=8a98fe5df3cf4207b7475fab13d8ed65]) successfully deleted
I20240502 21:27:01.554409 31381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a699f448e116428f85cfee6ccbc98612 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:01.557794 31393 catalog_manager.cc:5484] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d reported cstate change: config changed from index 3 to 4, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:01.562651 31536 raft_consensus.cc:1261] T a699f448e116428f85cfee6ccbc98612 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:01.563606 5157 consensus_queue.cc:1035] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:01.566910 31774 tablet_service.cc:1514] Processing DeleteTablet for tablet a699f448e116428f85cfee6ccbc98612 with delete_type TABLET_DATA_TOMBSTONED (TS 08d7a0680d244b55b93d17f9ab57b84c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51596
I20240502 21:27:01.568794 31536 raft_consensus.cc:2932] T a699f448e116428f85cfee6ccbc98612 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:27:01.572773 5170 tablet_replica.cc:330] T 5c302af336ed41aa8397f1b7479b6b7d P 1e04157b98dd4fad98858c290b73c3db: stopping tablet replica
I20240502 21:27:01.573006 5170 raft_consensus.cc:2229] T 5c302af336ed41aa8397f1b7479b6b7d P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:01.573186 5170 raft_consensus.cc:2258] T 5c302af336ed41aa8397f1b7479b6b7d P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:01.572408 5171 tablet_replica.cc:330] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c: stopping tablet replica
I20240502 21:27:01.573565 5171 raft_consensus.cc:2229] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:01.573870 5171 raft_consensus.cc:2258] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:01.574568 5170 ts_tablet_manager.cc:1875] T 5c302af336ed41aa8397f1b7479b6b7d P 1e04157b98dd4fad98858c290b73c3db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:01.574702 5171 ts_tablet_manager.cc:1875] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:01.580827 31393 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_extra_table_36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:01.582291 5171 ts_tablet_manager.cc:1888] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:01.582407 5171 log.cc:1196] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/wals/5c302af336ed41aa8397f1b7479b6b7d
I20240502 21:27:01.582444 5170 ts_tablet_manager.cc:1888] T 5c302af336ed41aa8397f1b7479b6b7d P 1e04157b98dd4fad98858c290b73c3db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:01.582652 5170 log.cc:1196] T 5c302af336ed41aa8397f1b7479b6b7d P 1e04157b98dd4fad98858c290b73c3db: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal/wals/5c302af336ed41aa8397f1b7479b6b7d
I20240502 21:27:01.582826 5171 ts_tablet_manager.cc:1909] T 5c302af336ed41aa8397f1b7479b6b7d P 08d7a0680d244b55b93d17f9ab57b84c: Deleting consensus metadata
I20240502 21:27:01.583830 31381 catalog_manager.cc:4830] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): tablet 5c302af336ed41aa8397f1b7479b6b7d (table rebalancer_dynamic_table_new_name [id=8a98fe5df3cf4207b7475fab13d8ed65]) successfully deleted
I20240502 21:27:01.584226 5171 tablet_replica.cc:330] T a699f448e116428f85cfee6ccbc98612 P 08d7a0680d244b55b93d17f9ab57b84c: stopping tablet replica
I20240502 21:27:01.584484 5171 raft_consensus.cc:2229] T a699f448e116428f85cfee6ccbc98612 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:01.584690 5171 raft_consensus.cc:2258] T a699f448e116428f85cfee6ccbc98612 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:01.586979 5171 ts_tablet_manager.cc:1875] T a699f448e116428f85cfee6ccbc98612 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:01.583207 5170 ts_tablet_manager.cc:1909] T 5c302af336ed41aa8397f1b7479b6b7d P 1e04157b98dd4fad98858c290b73c3db: Deleting consensus metadata
I20240502 21:27:01.588529 31381 catalog_manager.cc:4830] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): tablet 5c302af336ed41aa8397f1b7479b6b7d (table rebalancer_dynamic_table_new_name [id=8a98fe5df3cf4207b7475fab13d8ed65]) successfully deleted
I20240502 21:27:01.592564 5171 ts_tablet_manager.cc:1888] T a699f448e116428f85cfee6ccbc98612 P 08d7a0680d244b55b93d17f9ab57b84c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:27:01.593230 1945 tablet_service.cc:1467] Processing CreateTablet for tablet e4e50c50be16429faef86d47cc324bdb (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=57178ccbd1fe4c71a630c972457ec89e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:01.593405 31906 tablet_service.cc:1467] Processing CreateTablet for tablet e4e50c50be16429faef86d47cc324bdb (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=57178ccbd1fe4c71a630c972457ec89e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:01.593631 1945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4e50c50be16429faef86d47cc324bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.593770 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4e50c50be16429faef86d47cc324bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.593791 31514 tablet_service.cc:1467] Processing CreateTablet for tablet e4e50c50be16429faef86d47cc324bdb (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=57178ccbd1fe4c71a630c972457ec89e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:01.594197 31514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4e50c50be16429faef86d47cc324bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.596879 1814 tablet_service.cc:1467] Processing CreateTablet for tablet 69432f98b88740adb4fcbc112298cf20 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=57178ccbd1fe4c71a630c972457ec89e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:01.596887 1813 tablet_service.cc:1467] Processing CreateTablet for tablet 5d045ba2da9e482da25a39a0a2f38ff5 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=57178ccbd1fe4c71a630c972457ec89e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:01.597276 1814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69432f98b88740adb4fcbc112298cf20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.597465 1813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d045ba2da9e482da25a39a0a2f38ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.598059 1946 tablet_service.cc:1467] Processing CreateTablet for tablet 69432f98b88740adb4fcbc112298cf20 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=57178ccbd1fe4c71a630c972457ec89e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:01.598409 1946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69432f98b88740adb4fcbc112298cf20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.602466 31647 tablet_service.cc:1467] Processing CreateTablet for tablet 69432f98b88740adb4fcbc112298cf20 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=57178ccbd1fe4c71a630c972457ec89e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:01.602574 31645 tablet_service.cc:1467] Processing CreateTablet for tablet 5d045ba2da9e482da25a39a0a2f38ff5 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=57178ccbd1fe4c71a630c972457ec89e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:01.602900 31647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69432f98b88740adb4fcbc112298cf20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.603132 31516 tablet_service.cc:1467] Processing CreateTablet for tablet 5d045ba2da9e482da25a39a0a2f38ff5 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=57178ccbd1fe4c71a630c972457ec89e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:01.603435 31516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d045ba2da9e482da25a39a0a2f38ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.607645 5176 tablet_bootstrap.cc:492] T e4e50c50be16429faef86d47cc324bdb P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:27:01.608310 31645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d045ba2da9e482da25a39a0a2f38ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.610790 5176 tablet_bootstrap.cc:654] T e4e50c50be16429faef86d47cc324bdb P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.621618 5171 log.cc:1196] T a699f448e116428f85cfee6ccbc98612 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/wals/a699f448e116428f85cfee6ccbc98612
I20240502 21:27:01.622167 31797 raft_consensus.cc:1203] T f74ea10ab8a2459794f816707b87f494 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:01.626950 31381 catalog_manager.cc:4830] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): tablet a699f448e116428f85cfee6ccbc98612 (table rebalancer_extra_table_24 [id=da186563aba24f6d860086d2a27eeff3]) successfully deleted
I20240502 21:27:01.627336 5180 tablet_bootstrap.cc:492] T 69432f98b88740adb4fcbc112298cf20 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:27:01.628204 5179 tablet_bootstrap.cc:492] T 5d045ba2da9e482da25a39a0a2f38ff5 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:27:01.629940 5179 tablet_bootstrap.cc:654] T 5d045ba2da9e482da25a39a0a2f38ff5 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.630759 5180 tablet_bootstrap.cc:654] T 69432f98b88740adb4fcbc112298cf20 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.630795 5177 tablet_bootstrap.cc:492] T 69432f98b88740adb4fcbc112298cf20 P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:27:01.634881 5181 tablet_bootstrap.cc:492] T e4e50c50be16429faef86d47cc324bdb P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:27:01.635962 5177 tablet_bootstrap.cc:654] T 69432f98b88740adb4fcbc112298cf20 P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.637097 5181 tablet_bootstrap.cc:654] T e4e50c50be16429faef86d47cc324bdb P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.638768 31393 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:01.641304 5181 tablet_bootstrap.cc:492] T e4e50c50be16429faef86d47cc324bdb P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:27:01.641549 5181 ts_tablet_manager.cc:1367] T e4e50c50be16429faef86d47cc324bdb P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 21:27:01.642237 5181 raft_consensus.cc:348] T e4e50c50be16429faef86d47cc324bdb P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:01.642716 5181 raft_consensus.cc:374] T e4e50c50be16429faef86d47cc324bdb P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.642961 5181 raft_consensus.cc:729] T e4e50c50be16429faef86d47cc324bdb P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.643242 5181 consensus_queue.cc:260] T e4e50c50be16429faef86d47cc324bdb P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:01.645457 5179 tablet_bootstrap.cc:492] T 5d045ba2da9e482da25a39a0a2f38ff5 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:27:01.645666 5179 ts_tablet_manager.cc:1367] T 5d045ba2da9e482da25a39a0a2f38ff5 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20240502 21:27:01.645725 5181 ts_tablet_manager.cc:1398] T e4e50c50be16429faef86d47cc324bdb P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 21:27:01.646270 5179 raft_consensus.cc:348] T 5d045ba2da9e482da25a39a0a2f38ff5 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:01.646464 5179 raft_consensus.cc:374] T 5d045ba2da9e482da25a39a0a2f38ff5 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.646803 5179 raft_consensus.cc:729] T 5d045ba2da9e482da25a39a0a2f38ff5 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.647091 5179 consensus_queue.cc:260] T 5d045ba2da9e482da25a39a0a2f38ff5 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:01.647895 2208 tablet_service.cc:1467] Processing CreateTablet for tablet c490011176154e38b012cda0bbd695b2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0cd367f3616542169f8d3fc53d47e806]), partition=
I20240502 21:27:01.648041 5179 ts_tablet_manager.cc:1398] T 5d045ba2da9e482da25a39a0a2f38ff5 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:01.648247 2208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c490011176154e38b012cda0bbd695b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.648269 5179 tablet_bootstrap.cc:492] T 69432f98b88740adb4fcbc112298cf20 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:27:01.648865 1814 tablet_service.cc:1467] Processing CreateTablet for tablet c490011176154e38b012cda0bbd695b2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0cd367f3616542169f8d3fc53d47e806]), partition=
I20240502 21:27:01.649215 1814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c490011176154e38b012cda0bbd695b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.651177 5179 tablet_bootstrap.cc:654] T 69432f98b88740adb4fcbc112298cf20 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.652033 1946 tablet_service.cc:1467] Processing CreateTablet for tablet c490011176154e38b012cda0bbd695b2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0cd367f3616542169f8d3fc53d47e806]), partition=
I20240502 21:27:01.652390 1946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c490011176154e38b012cda0bbd695b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.669476 5180 tablet_bootstrap.cc:492] T 69432f98b88740adb4fcbc112298cf20 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:27:01.669695 5180 ts_tablet_manager.cc:1367] T 69432f98b88740adb4fcbc112298cf20 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.010s
I20240502 21:27:01.669718 5185 tablet_bootstrap.cc:492] T c490011176154e38b012cda0bbd695b2 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:27:01.670500 5180 raft_consensus.cc:348] T 69432f98b88740adb4fcbc112298cf20 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:01.670725 5180 raft_consensus.cc:374] T 69432f98b88740adb4fcbc112298cf20 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.670864 5180 raft_consensus.cc:729] T 69432f98b88740adb4fcbc112298cf20 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.671106 5180 consensus_queue.cc:260] T 69432f98b88740adb4fcbc112298cf20 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:01.671854 5180 ts_tablet_manager.cc:1398] T 69432f98b88740adb4fcbc112298cf20 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:27:01.673317 5180 tablet_bootstrap.cc:492] T 5d045ba2da9e482da25a39a0a2f38ff5 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:27:01.674844 5180 tablet_bootstrap.cc:654] T 5d045ba2da9e482da25a39a0a2f38ff5 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.674868 5185 tablet_bootstrap.cc:654] T c490011176154e38b012cda0bbd695b2 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.687206 5176 tablet_bootstrap.cc:492] T e4e50c50be16429faef86d47cc324bdb P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:27:01.687332 5176 ts_tablet_manager.cc:1367] T e4e50c50be16429faef86d47cc324bdb P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.080s user 0.013s sys 0.005s
I20240502 21:27:01.687703 5180 tablet_bootstrap.cc:492] T 5d045ba2da9e482da25a39a0a2f38ff5 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:27:01.687824 5180 ts_tablet_manager.cc:1367] T 5d045ba2da9e482da25a39a0a2f38ff5 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.006s
I20240502 21:27:01.688045 5176 raft_consensus.cc:348] T e4e50c50be16429faef86d47cc324bdb P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:01.688292 5176 raft_consensus.cc:374] T e4e50c50be16429faef86d47cc324bdb P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.688345 5176 raft_consensus.cc:729] T e4e50c50be16429faef86d47cc324bdb P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.688565 5176 consensus_queue.cc:260] T e4e50c50be16429faef86d47cc324bdb P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:01.689203 5176 ts_tablet_manager.cc:1398] T e4e50c50be16429faef86d47cc324bdb P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:01.689450 5176 tablet_bootstrap.cc:492] T 5d045ba2da9e482da25a39a0a2f38ff5 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:27:01.689697 5180 raft_consensus.cc:348] T 5d045ba2da9e482da25a39a0a2f38ff5 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:01.689882 5180 raft_consensus.cc:374] T 5d045ba2da9e482da25a39a0a2f38ff5 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.689929 5180 raft_consensus.cc:729] T 5d045ba2da9e482da25a39a0a2f38ff5 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.690089 5180 consensus_queue.cc:260] T 5d045ba2da9e482da25a39a0a2f38ff5 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:01.694123 5180 ts_tablet_manager.cc:1398] T 5d045ba2da9e482da25a39a0a2f38ff5 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 21:27:01.694882 5180 tablet_bootstrap.cc:492] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:27:01.696924 5180 tablet_bootstrap.cc:654] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.701545 5176 tablet_bootstrap.cc:654] T 5d045ba2da9e482da25a39a0a2f38ff5 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.703881 5177 tablet_bootstrap.cc:492] T 69432f98b88740adb4fcbc112298cf20 P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:27:01.704041 5177 ts_tablet_manager.cc:1367] T 69432f98b88740adb4fcbc112298cf20 P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.073s user 0.004s sys 0.012s
I20240502 21:27:01.705093 5177 raft_consensus.cc:348] T 69432f98b88740adb4fcbc112298cf20 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:01.706178 5177 raft_consensus.cc:374] T 69432f98b88740adb4fcbc112298cf20 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.706295 5177 raft_consensus.cc:729] T 69432f98b88740adb4fcbc112298cf20 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.706560 5177 consensus_queue.cc:260] T 69432f98b88740adb4fcbc112298cf20 P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:01.707819 4985 raft_consensus.cc:1050] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db: attempting to promote NON_VOTER 08d7a0680d244b55b93d17f9ab57b84c to VOTER
I20240502 21:27:01.708745 5177 ts_tablet_manager.cc:1398] T 69432f98b88740adb4fcbc112298cf20 P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:27:01.708616 4985 consensus_queue.cc:237] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } }
I20240502 21:27:01.709496 5177 tablet_bootstrap.cc:492] T e4e50c50be16429faef86d47cc324bdb P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:27:01.715602 5177 tablet_bootstrap.cc:654] T e4e50c50be16429faef86d47cc324bdb P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.717481 5179 tablet_bootstrap.cc:492] T 69432f98b88740adb4fcbc112298cf20 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:27:01.717948 5179 ts_tablet_manager.cc:1367] T 69432f98b88740adb4fcbc112298cf20 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.070s user 0.004s sys 0.010s
I20240502 21:27:01.719907 31797 raft_consensus.cc:1261] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEARNER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:01.720873 4986 consensus_queue.cc:1035] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:01.721520 4986 raft_consensus.cc:482] T 5d045ba2da9e482da25a39a0a2f38ff5 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:01.721688 4986 raft_consensus.cc:504] T 5d045ba2da9e482da25a39a0a2f38ff5 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:01.722486 4986 leader_election.cc:288] T 5d045ba2da9e482da25a39a0a2f38ff5 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:27:01.723459 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d045ba2da9e482da25a39a0a2f38ff5" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:27:01.725566 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d045ba2da9e482da25a39a0a2f38ff5" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:27:01.725821 31665 raft_consensus.cc:2454] T 5d045ba2da9e482da25a39a0a2f38ff5 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 0.
W20240502 21:27:01.726415 1770 leader_election.cc:341] T 5d045ba2da9e482da25a39a0a2f38ff5 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:01.736353 5176 tablet_bootstrap.cc:492] T 5d045ba2da9e482da25a39a0a2f38ff5 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:27:01.736766 5176 ts_tablet_manager.cc:1367] T 5d045ba2da9e482da25a39a0a2f38ff5 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.005s
I20240502 21:27:01.742476 1770 leader_election.cc:302] T 5d045ba2da9e482da25a39a0a2f38ff5 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 1e04157b98dd4fad98858c290b73c3db; no voters: a44f41b0b6644fb6aa29476aed8c964f
I20240502 21:27:01.743016 4744 raft_consensus.cc:2781] T 5d045ba2da9e482da25a39a0a2f38ff5 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:01.743119 4744 raft_consensus.cc:482] T 5d045ba2da9e482da25a39a0a2f38ff5 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:01.743237 4744 raft_consensus.cc:3037] T 5d045ba2da9e482da25a39a0a2f38ff5 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.764878 5176 raft_consensus.cc:348] T 5d045ba2da9e482da25a39a0a2f38ff5 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:01.765332 5176 raft_consensus.cc:374] T 5d045ba2da9e482da25a39a0a2f38ff5 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.765466 5176 raft_consensus.cc:729] T 5d045ba2da9e482da25a39a0a2f38ff5 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.765837 5176 consensus_queue.cc:260] T 5d045ba2da9e482da25a39a0a2f38ff5 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:01.768389 5176 ts_tablet_manager.cc:1398] T 5d045ba2da9e482da25a39a0a2f38ff5 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.032s user 0.002s sys 0.000s
I20240502 21:27:01.774451 4744 raft_consensus.cc:504] T 5d045ba2da9e482da25a39a0a2f38ff5 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:01.775908 4744 leader_election.cc:288] T 5d045ba2da9e482da25a39a0a2f38ff5 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Requested vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:27:01.777730 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d045ba2da9e482da25a39a0a2f38ff5" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:27:01.777966 31536 raft_consensus.cc:3037] T 5d045ba2da9e482da25a39a0a2f38ff5 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.781932 31536 raft_consensus.cc:2454] T 5d045ba2da9e482da25a39a0a2f38ff5 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:27:01.782591 1770 leader_election.cc:302] T 5d045ba2da9e482da25a39a0a2f38ff5 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:27:01.790854 5185 tablet_bootstrap.cc:492] T c490011176154e38b012cda0bbd695b2 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:27:01.791007 5185 ts_tablet_manager.cc:1367] T c490011176154e38b012cda0bbd695b2 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.121s user 0.006s sys 0.006s
I20240502 21:27:01.791815 5185 raft_consensus.cc:348] T c490011176154e38b012cda0bbd695b2 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:27:01.792065 5185 raft_consensus.cc:374] T c490011176154e38b012cda0bbd695b2 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.792104 5089 raft_consensus.cc:482] T e4e50c50be16429faef86d47cc324bdb P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:01.792163 5185 raft_consensus.cc:729] T c490011176154e38b012cda0bbd695b2 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.792234 5089 raft_consensus.cc:504] T e4e50c50be16429faef86d47cc324bdb P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:01.792405 5185 consensus_queue.cc:260] T c490011176154e38b012cda0bbd695b2 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:27:01.793047 5089 leader_election.cc:288] T e4e50c50be16429faef86d47cc324bdb P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:27:01.793143 5185 ts_tablet_manager.cc:1398] T c490011176154e38b012cda0bbd695b2 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:01.793680 1962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e50c50be16429faef86d47cc324bdb" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
W20240502 21:27:01.794167 31469 leader_election.cc:341] T e4e50c50be16429faef86d47cc324bdb P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:01.796489 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d045ba2da9e482da25a39a0a2f38ff5" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:27:01.796694 31665 raft_consensus.cc:3037] T 5d045ba2da9e482da25a39a0a2f38ff5 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.799495 31665 raft_consensus.cc:2454] T 5d045ba2da9e482da25a39a0a2f38ff5 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
W20240502 21:27:01.804677 31600 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.226.134:43753, user_credentials={real_user=slave}} blocked reactor thread for 63684.6us
I20240502 21:27:01.805858 2225 raft_consensus.cc:1203] T ae56d18c629a4e5eb50911c89de4227c P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:27:01.828117 4801 raft_consensus.cc:482] T 69432f98b88740adb4fcbc112298cf20 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:01.828274 4801 raft_consensus.cc:504] T 69432f98b88740adb4fcbc112298cf20 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:01.829135 4801 leader_election.cc:288] T 69432f98b88740adb4fcbc112298cf20 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:27:01.829689 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69432f98b88740adb4fcbc112298cf20" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
W20240502 21:27:01.830179 1902 leader_election.cc:341] T 69432f98b88740adb4fcbc112298cf20 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:01.830698 1834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69432f98b88740adb4fcbc112298cf20" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:27:01.830896 1834 raft_consensus.cc:2454] T 69432f98b88740adb4fcbc112298cf20 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 0.
I20240502 21:27:01.831300 1901 leader_election.cc:302] T 69432f98b88740adb4fcbc112298cf20 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, 9dec645b473f40578e726f3e50e219bf; no voters: 05f4fac41c4f4959b9b316cb112d0ada
I20240502 21:27:01.831761 4801 raft_consensus.cc:2781] T 69432f98b88740adb4fcbc112298cf20 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:01.831853 4801 raft_consensus.cc:482] T 69432f98b88740adb4fcbc112298cf20 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:01.831939 4801 raft_consensus.cc:3037] T 69432f98b88740adb4fcbc112298cf20 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.835140 4801 raft_consensus.cc:504] T 69432f98b88740adb4fcbc112298cf20 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:01.835512 5177 tablet_bootstrap.cc:492] T e4e50c50be16429faef86d47cc324bdb P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:27:01.835687 5177 ts_tablet_manager.cc:1367] T e4e50c50be16429faef86d47cc324bdb P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.126s user 0.024s sys 0.033s
I20240502 21:27:01.836236 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69432f98b88740adb4fcbc112298cf20" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
W20240502 21:27:01.836692 1902 leader_election.cc:341] T 69432f98b88740adb4fcbc112298cf20 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:01.836961 1834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69432f98b88740adb4fcbc112298cf20" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:27:01.837149 1834 raft_consensus.cc:3037] T 69432f98b88740adb4fcbc112298cf20 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.838531 5212 raft_consensus.cc:2781] T 5d045ba2da9e482da25a39a0a2f38ff5 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:01.838647 5212 raft_consensus.cc:686] T 5d045ba2da9e482da25a39a0a2f38ff5 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Becoming Leader. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Running, Role: LEADER
I20240502 21:27:01.838909 5212 consensus_queue.cc:237] T 5d045ba2da9e482da25a39a0a2f38ff5 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:01.839591 1834 raft_consensus.cc:2454] T 69432f98b88740adb4fcbc112298cf20 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 1.
I20240502 21:27:01.839900 5177 raft_consensus.cc:348] T e4e50c50be16429faef86d47cc324bdb P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:01.840121 5177 raft_consensus.cc:374] T e4e50c50be16429faef86d47cc324bdb P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.840220 5177 raft_consensus.cc:729] T e4e50c50be16429faef86d47cc324bdb P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.840121 1901 leader_election.cc:302] T 69432f98b88740adb4fcbc112298cf20 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, 9dec645b473f40578e726f3e50e219bf; no voters: 05f4fac41c4f4959b9b316cb112d0ada
I20240502 21:27:01.840513 3818 raft_consensus.cc:2781] T 69432f98b88740adb4fcbc112298cf20 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:01.840689 3818 raft_consensus.cc:686] T 69432f98b88740adb4fcbc112298cf20 P 9dec645b473f40578e726f3e50e219bf [term 1 LEADER]: Becoming Leader. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Running, Role: LEADER
I20240502 21:27:01.841017 3818 consensus_queue.cc:237] T 69432f98b88740adb4fcbc112298cf20 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:01.845603 5180 tablet_bootstrap.cc:492] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:27:01.845849 5179 raft_consensus.cc:348] T 69432f98b88740adb4fcbc112298cf20 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:01.846220 5180 ts_tablet_manager.cc:1367] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.151s user 0.004s sys 0.002s
I20240502 21:27:01.846343 5179 raft_consensus.cc:374] T 69432f98b88740adb4fcbc112298cf20 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.847251 5179 raft_consensus.cc:729] T 69432f98b88740adb4fcbc112298cf20 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.847853 5179 consensus_queue.cc:260] T 69432f98b88740adb4fcbc112298cf20 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:01.848647 5179 ts_tablet_manager.cc:1398] T 69432f98b88740adb4fcbc112298cf20 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.130s user 0.001s sys 0.001s
I20240502 21:27:01.848969 5180 raft_consensus.cc:348] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:27:01.852546 5180 raft_consensus.cc:374] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.852869 5180 raft_consensus.cc:729] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.853166 5180 consensus_queue.cc:260] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:27:01.853128 31389 catalog_manager.cc:5484] T 69432f98b88740adb4fcbc112298cf20 P 9dec645b473f40578e726f3e50e219bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dec645b473f40578e726f3e50e219bf (127.24.226.134). New cstate: current_term: 1 leader_uuid: "9dec645b473f40578e726f3e50e219bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:01.855131 2092 raft_consensus.cc:1261] T 9bc57b8a0388416290997df544e5cc0b P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:01.855196 5180 ts_tablet_manager.cc:1398] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240502 21:27:01.858316 5205 consensus_queue.cc:1035] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:01.858671 4801 leader_election.cc:288] T 69432f98b88740adb4fcbc112298cf20 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 election: Requested vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:27:01.861716 31923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e50c50be16429faef86d47cc324bdb" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:27:01.861912 31923 raft_consensus.cc:2454] T e4e50c50be16429faef86d47cc324bdb P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:27:01.862722 31472 leader_election.cc:302] T e4e50c50be16429faef86d47cc324bdb P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, a44f41b0b6644fb6aa29476aed8c964f; no voters: 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:01.863415 4942 raft_consensus.cc:2781] T e4e50c50be16429faef86d47cc324bdb P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:01.864758 1962 raft_consensus.cc:1261] T 9bc57b8a0388416290997df544e5cc0b P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:01.863552 4942 raft_consensus.cc:482] T e4e50c50be16429faef86d47cc324bdb P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:01.864892 4942 raft_consensus.cc:3037] T e4e50c50be16429faef86d47cc324bdb P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.865554 4744 consensus_queue.cc:1035] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:01.867381 4942 raft_consensus.cc:504] T e4e50c50be16429faef86d47cc324bdb P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:01.869875 4942 leader_election.cc:288] T e4e50c50be16429faef86d47cc324bdb P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Requested vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:27:01.870128 1962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e50c50be16429faef86d47cc324bdb" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf"
I20240502 21:27:01.840520 5177 consensus_queue.cc:260] T e4e50c50be16429faef86d47cc324bdb P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:01.871163 31923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e50c50be16429faef86d47cc324bdb" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:27:01.871294 31923 raft_consensus.cc:3037] T e4e50c50be16429faef86d47cc324bdb P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.872536 1962 raft_consensus.cc:3037] T e4e50c50be16429faef86d47cc324bdb P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.873889 31923 raft_consensus.cc:2454] T e4e50c50be16429faef86d47cc324bdb P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:27:01.874579 31472 leader_election.cc:302] T e4e50c50be16429faef86d47cc324bdb P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:27:01.875007 5089 raft_consensus.cc:2781] T e4e50c50be16429faef86d47cc324bdb P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:01.875180 5089 raft_consensus.cc:686] T e4e50c50be16429faef86d47cc324bdb P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:27:01.875253 1962 raft_consensus.cc:2454] T e4e50c50be16429faef86d47cc324bdb P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:27:01.875437 5089 consensus_queue.cc:237] T e4e50c50be16429faef86d47cc324bdb P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:01.878494 5177 ts_tablet_manager.cc:1398] T e4e50c50be16429faef86d47cc324bdb P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.043s user 0.000s sys 0.002s
I20240502 21:27:01.878194 2094 raft_consensus.cc:2932] T 9bc57b8a0388416290997df544e5cc0b P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:27:01.878770 5177 tablet_bootstrap.cc:492] T c490011176154e38b012cda0bbd695b2 P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:27:01.880239 5177 tablet_bootstrap.cc:654] T c490011176154e38b012cda0bbd695b2 P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:01.887130 4985 raft_consensus.cc:2932] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:27:01.887853 31389 catalog_manager.cc:5484] T 9bc57b8a0388416290997df544e5cc0b P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: config changed from index 2 to 3, 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:27:01.903996 5177 tablet_bootstrap.cc:492] T c490011176154e38b012cda0bbd695b2 P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:27:01.903584 31797 raft_consensus.cc:2932] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:27:01.904153 5177 ts_tablet_manager.cc:1367] T c490011176154e38b012cda0bbd695b2 P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.001s
I20240502 21:27:01.905001 5177 raft_consensus.cc:348] T c490011176154e38b012cda0bbd695b2 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:27:01.905196 5177 raft_consensus.cc:374] T c490011176154e38b012cda0bbd695b2 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:01.905283 5177 raft_consensus.cc:729] T c490011176154e38b012cda0bbd695b2 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:27:01.905853 31389 catalog_manager.cc:5484] T e4e50c50be16429faef86d47cc324bdb P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 0 to 1, leader changed from <none> to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:01.906190 5177 consensus_queue.cc:260] T c490011176154e38b012cda0bbd695b2 P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:27:01.907737 5177 ts_tablet_manager.cc:1398] T c490011176154e38b012cda0bbd695b2 P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:01.910805 5218 raft_consensus.cc:482] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:01.911077 5218 raft_consensus.cc:504] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:27:01.912168 5218 leader_election.cc:288] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:27:01.913028 2225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c490011176154e38b012cda0bbd695b2" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:27:01.913254 1962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c490011176154e38b012cda0bbd695b2" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:27:01.913314 2225 raft_consensus.cc:2454] T c490011176154e38b012cda0bbd695b2 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 0.
I20240502 21:27:01.913452 1962 raft_consensus.cc:2454] T c490011176154e38b012cda0bbd695b2 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 0.
I20240502 21:27:01.913892 1767 leader_election.cc:302] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:27:01.914564 5215 raft_consensus.cc:2781] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:01.914681 5215 raft_consensus.cc:482] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:01.914820 5215 raft_consensus.cc:3037] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.917155 31393 catalog_manager.cc:5484] T 5d045ba2da9e482da25a39a0a2f38ff5 P 1e04157b98dd4fad98858c290b73c3db reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e04157b98dd4fad98858c290b73c3db (127.24.226.133). New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:01.919409 5215 raft_consensus.cc:504] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:27:01.921448 1962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c490011176154e38b012cda0bbd695b2" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf"
I20240502 21:27:01.921634 1962 raft_consensus.cc:3037] T c490011176154e38b012cda0bbd695b2 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.921617 2225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c490011176154e38b012cda0bbd695b2" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:27:01.921866 2225 raft_consensus.cc:3037] T c490011176154e38b012cda0bbd695b2 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:01.924178 1962 raft_consensus.cc:2454] T c490011176154e38b012cda0bbd695b2 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:27:01.925140 2225 raft_consensus.cc:2454] T c490011176154e38b012cda0bbd695b2 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:27:01.925608 1767 leader_election.cc:302] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:27:01.927301 5157 raft_consensus.cc:1050] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d: attempting to promote NON_VOTER 9dec645b473f40578e726f3e50e219bf to VOTER
I20240502 21:27:01.928417 5157 consensus_queue.cc:237] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:27:01.930902 5205 raft_consensus.cc:2781] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:01.931059 5205 raft_consensus.cc:686] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Becoming Leader. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Running, Role: LEADER
I20240502 21:27:01.931907 1962 raft_consensus.cc:2932] T 9bc57b8a0388416290997df544e5cc0b P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:27:01.932262 5205 consensus_queue.cc:237] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:27:01.939838 1962 raft_consensus.cc:1261] T 59f36d947f4341f39b6969cc1e065b6c P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:01.939924 31923 raft_consensus.cc:1261] T 59f36d947f4341f39b6969cc1e065b6c P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:01.940022 2094 raft_consensus.cc:1261] T 59f36d947f4341f39b6969cc1e065b6c P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:01.940526 5156 consensus_queue.cc:1035] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:01.940819 5157 consensus_queue.cc:1035] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:01.943053 5153 consensus_queue.cc:1035] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:01.955638 31389 catalog_manager.cc:5484] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e04157b98dd4fad98858c290b73c3db (127.24.226.133). New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:01.959043 5157 raft_consensus.cc:2932] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:27:01.961259 1962 raft_consensus.cc:2932] T 59f36d947f4341f39b6969cc1e065b6c P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:27:01.966132 31923 raft_consensus.cc:2932] T 59f36d947f4341f39b6969cc1e065b6c P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:27:01.968833 5215 leader_election.cc:288] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Requested vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:27:01.975641 2094 raft_consensus.cc:2932] T 59f36d947f4341f39b6969cc1e065b6c P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:27:01.977512 31393 catalog_manager.cc:5484] T 59f36d947f4341f39b6969cc1e065b6c P 9dec645b473f40578e726f3e50e219bf reported cstate change: config changed from index 2 to 3, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:27:01.990976 2225 consensus_queue.cc:237] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:27:01.994355 31923 raft_consensus.cc:1261] T 59f36d947f4341f39b6969cc1e065b6c P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:01.994886 1962 raft_consensus.cc:1261] T 59f36d947f4341f39b6969cc1e065b6c P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:01.995131 5159 consensus_queue.cc:1035] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:01.997128 5159 consensus_queue.cc:1035] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:01.999516 5159 raft_consensus.cc:2932] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:27:02.008548 1964 raft_consensus.cc:2932] T 59f36d947f4341f39b6969cc1e065b6c P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:27:02.014772 31381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 59f36d947f4341f39b6969cc1e065b6c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:02.015339 31923 raft_consensus.cc:2932] T 59f36d947f4341f39b6969cc1e065b6c P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:27:02.020265 31392 catalog_manager.cc:5484] T 59f36d947f4341f39b6969cc1e065b6c P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:27:02.025032 2077 tablet_service.cc:1514] Processing DeleteTablet for tablet 59f36d947f4341f39b6969cc1e065b6c with delete_type TABLET_DATA_TOMBSTONED (TS 1ade0354c44c4c37ae480c7a94b11e1b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36740
I20240502 21:27:02.032680 5233 tablet_replica.cc:330] T 59f36d947f4341f39b6969cc1e065b6c P 1ade0354c44c4c37ae480c7a94b11e1b: stopping tablet replica
I20240502 21:27:02.033033 5233 raft_consensus.cc:2229] T 59f36d947f4341f39b6969cc1e065b6c P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:02.034041 5233 raft_consensus.cc:2258] T 59f36d947f4341f39b6969cc1e065b6c P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:02.035418 5233 ts_tablet_manager.cc:1875] T 59f36d947f4341f39b6969cc1e065b6c P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:02.041371 5233 ts_tablet_manager.cc:1888] T 59f36d947f4341f39b6969cc1e065b6c P 1ade0354c44c4c37ae480c7a94b11e1b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:27:02.041554 5233 log.cc:1196] T 59f36d947f4341f39b6969cc1e065b6c P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal/wals/59f36d947f4341f39b6969cc1e065b6c
I20240502 21:27:02.042436 31381 catalog_manager.cc:4830] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): tablet 59f36d947f4341f39b6969cc1e065b6c (table rebalancer_extra_table_12 [id=daf389fa76d84011ad57fe7156cfa6a8]) successfully deleted
I20240502 21:27:02.059530 5090 raft_consensus.cc:1050] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f: attempting to promote NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b to VOTER
I20240502 21:27:02.060171 5090 consensus_queue.cc:237] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } }
I20240502 21:27:02.068933 3818 raft_consensus.cc:1050] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf: attempting to promote NON_VOTER a44f41b0b6644fb6aa29476aed8c964f to VOTER
I20240502 21:27:02.069587 31386 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:02.069633 3818 consensus_queue.cc:237] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } }
I20240502 21:27:02.079476 1964 raft_consensus.cc:1261] T c490011176154e38b012cda0bbd695b2 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:02.079809 2225 raft_consensus.cc:1261] T c490011176154e38b012cda0bbd695b2 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:02.080116 5223 consensus_queue.cc:1035] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:02.080732 31536 raft_consensus.cc:1261] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEARNER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:02.081388 31665 raft_consensus.cc:1261] T 51551a1967b74094b373cc030e2bf850 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:02.081329 5223 consensus_queue.cc:1035] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:02.082221 3818 consensus_queue.cc:1035] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:02.082561 4801 consensus_queue.cc:1035] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:02.083333 2225 raft_consensus.cc:1261] T 51551a1967b74094b373cc030e2bf850 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:02.084275 4801 consensus_queue.cc:1035] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:02.089875 5134 raft_consensus.cc:2932] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:27:02.093811 2225 raft_consensus.cc:2932] T 51551a1967b74094b373cc030e2bf850 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:27:02.097011 31665 raft_consensus.cc:2932] T 51551a1967b74094b373cc030e2bf850 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:27:02.103768 31386 catalog_manager.cc:5484] T 51551a1967b74094b373cc030e2bf850 P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: config changed from index 2 to 3, a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9dec645b473f40578e726f3e50e219bf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:27:02.106945 5069 raft_consensus.cc:482] T 69432f98b88740adb4fcbc112298cf20 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:02.107092 5069 raft_consensus.cc:504] T 69432f98b88740adb4fcbc112298cf20 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:02.107724 5069 leader_election.cc:288] T 69432f98b88740adb4fcbc112298cf20 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:27:02.108179 31389 catalog_manager.cc:5576] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753) does not have the latest schema for tablet c490011176154e38b012cda0bbd695b2 (table rebalancer_dynamic_table [id=0cd367f3616542169f8d3fc53d47e806]). Expected version 1 got 0
I20240502 21:27:02.108543 1964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69432f98b88740adb4fcbc112298cf20" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:27:02.110067 1834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69432f98b88740adb4fcbc112298cf20" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:27:02.110321 1834 raft_consensus.cc:2379] T 69432f98b88740adb4fcbc112298cf20 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 05f4fac41c4f4959b9b316cb112d0ada in current term 1: Already voted for candidate 9dec645b473f40578e726f3e50e219bf in this term.
I20240502 21:27:02.111301 31602 leader_election.cc:302] T 69432f98b88740adb4fcbc112298cf20 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada; no voters: 1e04157b98dd4fad98858c290b73c3db, 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:02.112421 5069 raft_consensus.cc:3037] T 69432f98b88740adb4fcbc112298cf20 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:02.114595 5069 raft_consensus.cc:2726] T 69432f98b88740adb4fcbc112298cf20 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:27:02.120767 5259 tablet.cc:1722] T c490011176154e38b012cda0bbd695b2 P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:02.121024 5258 tablet.cc:1722] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:02.121210 31392 catalog_manager.cc:5576] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753) does not have the latest schema for tablet c490011176154e38b012cda0bbd695b2 (table rebalancer_dynamic_table [id=0cd367f3616542169f8d3fc53d47e806]). Expected version 1 got 0
I20240502 21:27:02.121565 5258 tablet.cc:1620] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db: MemRowSet was empty: no flush needed.
I20240502 21:27:02.121726 31386 catalog_manager.cc:5576] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863) does not have the latest schema for tablet c490011176154e38b012cda0bbd695b2 (table rebalancer_dynamic_table [id=0cd367f3616542169f8d3fc53d47e806]). Expected version 1 got 0
I20240502 21:27:02.122356 5259 tablet.cc:1620] T c490011176154e38b012cda0bbd695b2 P 9dec645b473f40578e726f3e50e219bf: MemRowSet was empty: no flush needed.
I20240502 21:27:02.133293 31392 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=0cd367f3616542169f8d3fc53d47e806] alter complete (version 1)
I20240502 21:27:02.139645 31536 raft_consensus.cc:2932] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:27:02.142010 1964 consensus_queue.cc:237] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } }
I20240502 21:27:02.142980 31386 catalog_manager.cc:5576] TS d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907) does not have the latest schema for tablet c490011176154e38b012cda0bbd695b2 (table rebalancer_dynamic_table [id=0cd367f3616542169f8d3fc53d47e806]). Expected version 1 got 0
I20240502 21:27:02.144747 2230 raft_consensus.cc:1261] T 51551a1967b74094b373cc030e2bf850 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:02.145661 5134 consensus_queue.cc:1035] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:02.146052 31536 raft_consensus.cc:1261] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 9dec645b473f40578e726f3e50e219bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:02.148304 3818 raft_consensus.cc:2932] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:27:02.148689 2094 raft_consensus.cc:1261] T dd0198178da640688230aeb5a1106cb4 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:02.149561 31923 raft_consensus.cc:1261] T dd0198178da640688230aeb5a1106cb4 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:02.150066 31797 raft_consensus.cc:1261] T dd0198178da640688230aeb5a1106cb4 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:02.150171 5090 consensus_queue.cc:1035] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:02.150440 5089 consensus_queue.cc:1035] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:02.151384 4801 consensus_queue.cc:1035] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:02.151641 2230 raft_consensus.cc:2932] T 51551a1967b74094b373cc030e2bf850 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:27:02.153455 4942 consensus_queue.cc:1035] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:02.156090 31379 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 51551a1967b74094b373cc030e2bf850 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:02.159139 31392 catalog_manager.cc:5484] T 51551a1967b74094b373cc030e2bf850 P 9dec645b473f40578e726f3e50e219bf reported cstate change: config changed from index 3 to 4, VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) evicted. New cstate: current_term: 1 leader_uuid: "9dec645b473f40578e726f3e50e219bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:02.160465 5261 tablet.cc:1722] T c490011176154e38b012cda0bbd695b2 P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:02.161007 5261 tablet.cc:1620] T c490011176154e38b012cda0bbd695b2 P d1406e92aec84ac585304e57bd37a40d: MemRowSet was empty: no flush needed.
I20240502 21:27:02.163859 5090 raft_consensus.cc:2932] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:27:02.165879 31797 raft_consensus.cc:2932] T dd0198178da640688230aeb5a1106cb4 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:27:02.169042 2094 raft_consensus.cc:2932] T dd0198178da640688230aeb5a1106cb4 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:27:02.171759 31386 catalog_manager.cc:5576] TS d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907) does not have the latest schema for tablet c490011176154e38b012cda0bbd695b2 (table rebalancer_dynamic_table [id=0cd367f3616542169f8d3fc53d47e806]). Expected version 1 got 0
I20240502 21:27:02.173030 31647 tablet_service.cc:1514] Processing DeleteTablet for tablet 51551a1967b74094b373cc030e2bf850 with delete_type TABLET_DATA_TOMBSTONED (TS 05f4fac41c4f4959b9b316cb112d0ada not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52130
I20240502 21:27:02.175252 5265 tablet_replica.cc:330] T 51551a1967b74094b373cc030e2bf850 P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:27:02.175135 31536 raft_consensus.cc:2932] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } } }
I20240502 21:27:02.176005 5265 raft_consensus.cc:2229] T 51551a1967b74094b373cc030e2bf850 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:02.176898 31389 catalog_manager.cc:5484] T dd0198178da640688230aeb5a1106cb4 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:27:02.178813 5265 raft_consensus.cc:2258] T 51551a1967b74094b373cc030e2bf850 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:02.180074 5265 ts_tablet_manager.cc:1875] T 51551a1967b74094b373cc030e2bf850 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:02.191484 4904 raft_consensus.cc:1050] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b: attempting to promote NON_VOTER 08d7a0680d244b55b93d17f9ab57b84c to VOTER
I20240502 21:27:02.191190 31923 raft_consensus.cc:2932] T dd0198178da640688230aeb5a1106cb4 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:27:02.192207 4904 consensus_queue.cc:237] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } }
I20240502 21:27:02.193806 5265 ts_tablet_manager.cc:1888] T 51551a1967b74094b373cc030e2bf850 P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:27:02.193960 5265 log.cc:1196] T 51551a1967b74094b373cc030e2bf850 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/51551a1967b74094b373cc030e2bf850
I20240502 21:27:02.194846 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet 51551a1967b74094b373cc030e2bf850 (table rebalancer_extra_table_16 [id=cc412415870b4447b2e9707c2f7fd323]) successfully deleted
I20240502 21:27:02.208984 31797 raft_consensus.cc:1261] T f74ea10ab8a2459794f816707b87f494 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEARNER]: Refusing update from remote peer 1ade0354c44c4c37ae480c7a94b11e1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:02.210072 4904 consensus_queue.cc:1035] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:02.210161 1834 raft_consensus.cc:1261] T f74ea10ab8a2459794f816707b87f494 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer 1ade0354c44c4c37ae480c7a94b11e1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:02.210906 2230 raft_consensus.cc:1261] T f74ea10ab8a2459794f816707b87f494 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 1ade0354c44c4c37ae480c7a94b11e1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:02.213869 4905 consensus_queue.cc:1035] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:02.214408 4905 consensus_queue.cc:1035] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:02.218853 3818 consensus_queue.cc:1035] T 69432f98b88740adb4fcbc112298cf20 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:02.233969 4905 raft_consensus.cc:2932] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:27:02.237180 31386 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:02.242996 31392 catalog_manager.cc:5484] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: config changed from index 2 to 3, 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:02.250438 5258 tablet.cc:1722] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:02.250927 5258 tablet.cc:1620] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db: MemRowSet was empty: no flush needed.
I20240502 21:27:02.252553 2094 consensus_queue.cc:237] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } }
I20240502 21:27:02.258597 5134 consensus_queue.cc:1035] T 69432f98b88740adb4fcbc112298cf20 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:02.259678 5261 tablet.cc:1722] T c490011176154e38b012cda0bbd695b2 P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:02.260087 5261 tablet.cc:1620] T c490011176154e38b012cda0bbd695b2 P d1406e92aec84ac585304e57bd37a40d: MemRowSet was empty: no flush needed.
I20240502 21:27:02.264755 5259 tablet.cc:1722] T c490011176154e38b012cda0bbd695b2 P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:02.265199 5259 tablet.cc:1620] T c490011176154e38b012cda0bbd695b2 P 9dec645b473f40578e726f3e50e219bf: MemRowSet was empty: no flush needed.
I20240502 21:27:02.265859 31386 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=0cd367f3616542169f8d3fc53d47e806] alter complete (version 2)
I20240502 21:27:02.268317 5090 consensus_queue.cc:1035] T e4e50c50be16429faef86d47cc324bdb P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:02.279783 31797 raft_consensus.cc:2932] T f74ea10ab8a2459794f816707b87f494 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:27:02.282586 4942 consensus_queue.cc:1035] T e4e50c50be16429faef86d47cc324bdb P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:02.288486 1833 raft_consensus.cc:2932] T f74ea10ab8a2459794f816707b87f494 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:27:02.298945 31797 raft_consensus.cc:1261] T f74ea10ab8a2459794f816707b87f494 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 1ade0354c44c4c37ae480c7a94b11e1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:02.300688 1833 raft_consensus.cc:1261] T f74ea10ab8a2459794f816707b87f494 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer 1ade0354c44c4c37ae480c7a94b11e1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:02.301856 4904 consensus_queue.cc:1035] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:02.311043 4904 consensus_queue.cc:1035] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:02.314738 5019 raft_consensus.cc:2932] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:27:02.319170 31381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f74ea10ab8a2459794f816707b87f494 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:02.323195 31386 catalog_manager.cc:5484] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: config changed from index 3 to 4, VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) evicted. New cstate: current_term: 1 leader_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:02.324376 1833 raft_consensus.cc:2932] T f74ea10ab8a2459794f816707b87f494 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:27:02.327025 2208 tablet_service.cc:1514] Processing DeleteTablet for tablet f74ea10ab8a2459794f816707b87f494 with delete_type TABLET_DATA_TOMBSTONED (TS d1406e92aec84ac585304e57bd37a40d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36984
I20240502 21:27:02.327631 5284 tablet_replica.cc:330] T f74ea10ab8a2459794f816707b87f494 P d1406e92aec84ac585304e57bd37a40d: stopping tablet replica
I20240502 21:27:02.327879 5284 raft_consensus.cc:2229] T f74ea10ab8a2459794f816707b87f494 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:02.328091 5284 pending_rounds.cc:62] T f74ea10ab8a2459794f816707b87f494 P d1406e92aec84ac585304e57bd37a40d: Trying to abort 1 pending ops.
I20240502 21:27:02.328280 5284 pending_rounds.cc:68] T f74ea10ab8a2459794f816707b87f494 P d1406e92aec84ac585304e57bd37a40d: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7023350670097399808 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "f74ea10ab8a2459794f816707b87f494" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } } }
I20240502 21:27:02.328672 5284 raft_consensus.cc:2894] T f74ea10ab8a2459794f816707b87f494 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240502 21:27:02.328828 5284 raft_consensus.cc:2866] T f74ea10ab8a2459794f816707b87f494 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 21:27:02.329010 5284 raft_consensus.cc:2258] T f74ea10ab8a2459794f816707b87f494 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:02.329743 5284 ts_tablet_manager.cc:1875] T f74ea10ab8a2459794f816707b87f494 P d1406e92aec84ac585304e57bd37a40d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:02.337281 5284 ts_tablet_manager.cc:1888] T f74ea10ab8a2459794f816707b87f494 P d1406e92aec84ac585304e57bd37a40d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:27:02.337448 5284 log.cc:1196] T f74ea10ab8a2459794f816707b87f494 P d1406e92aec84ac585304e57bd37a40d: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal/wals/f74ea10ab8a2459794f816707b87f494
I20240502 21:27:02.338748 31381 catalog_manager.cc:4830] TS d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): tablet f74ea10ab8a2459794f816707b87f494 (table rebalancer_extra_table_2 [id=408b4cadaf474250a5666b6a49f4f8cc]) successfully deleted
I20240502 21:27:02.332895 31797 raft_consensus.cc:2932] T f74ea10ab8a2459794f816707b87f494 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1406e92aec84ac585304e57bd37a40d (127.24.226.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:27:02.370800 1833 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2c737c49a32b4f3ea5ef735fa254d48e"
dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59754
I20240502 21:27:02.371066 1833 raft_consensus.cc:595] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Received request to transfer leadership
I20240502 21:27:02.371049 31392 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:02.380570 5258 tablet.cc:1722] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:02.381788 5259 tablet.cc:1722] T c490011176154e38b012cda0bbd695b2 P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:02.393061 5261 tablet.cc:1722] T c490011176154e38b012cda0bbd695b2 P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:02.398185 31392 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table_new_name [id=0cd367f3616542169f8d3fc53d47e806] alter complete (version 3)
I20240502 21:27:02.405054 5202 raft_consensus.cc:1050] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1: attempting to promote NON_VOTER d1406e92aec84ac585304e57bd37a40d to VOTER
I20240502 21:27:02.405885 5202 consensus_queue.cc:237] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } }
I20240502 21:27:02.408252 2230 raft_consensus.cc:1261] T ae56d18c629a4e5eb50911c89de4227c P d1406e92aec84ac585304e57bd37a40d [term 1 LEARNER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:02.408231 1964 raft_consensus.cc:1261] T ae56d18c629a4e5eb50911c89de4227c P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:02.408612 31665 raft_consensus.cc:1261] T ae56d18c629a4e5eb50911c89de4227c P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:02.408932 5207 consensus_queue.cc:1035] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:02.409165 5203 consensus_queue.cc:1035] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:02.409628 5203 consensus_queue.cc:1035] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:02.414664 5219 consensus_queue.cc:1035] T 5d045ba2da9e482da25a39a0a2f38ff5 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:02.420558 5215 consensus_queue.cc:1035] T 5d045ba2da9e482da25a39a0a2f38ff5 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:02.420367 5207 raft_consensus.cc:2932] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:27:02.420890 2230 raft_consensus.cc:2932] T ae56d18c629a4e5eb50911c89de4227c P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:27:02.429688 31665 raft_consensus.cc:2932] T ae56d18c629a4e5eb50911c89de4227c P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:27:02.430964 31392 catalog_manager.cc:5484] T ae56d18c629a4e5eb50911c89de4227c P d1406e92aec84ac585304e57bd37a40d reported cstate change: config changed from index 7 to 8, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:27:02.432770 31389 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_extra_table_38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:02.434345 1964 raft_consensus.cc:2932] T ae56d18c629a4e5eb50911c89de4227c P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d1406e92aec84ac585304e57bd37a40d (127.24.226.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:27:02.439086 31923 consensus_queue.cc:237] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } }
I20240502 21:27:02.441813 1964 raft_consensus.cc:1261] T ae56d18c629a4e5eb50911c89de4227c P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:02.442126 2230 raft_consensus.cc:1261] T ae56d18c629a4e5eb50911c89de4227c P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:02.442204 5303 consensus_queue.cc:1035] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:02.442831 5202 consensus_queue.cc:1035] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:02.445612 2230 raft_consensus.cc:2932] T ae56d18c629a4e5eb50911c89de4227c P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:27:02.447638 1965 raft_consensus.cc:2932] T ae56d18c629a4e5eb50911c89de4227c P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:27:02.453943 31388 catalog_manager.cc:5484] T ae56d18c629a4e5eb50911c89de4227c P 9dec645b473f40578e726f3e50e219bf reported cstate change: config changed from index 8 to 9, VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) evicted. New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:27:02.445209 5207 raft_consensus.cc:2932] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } }
I20240502 21:27:02.458047 31647 tablet_service.cc:1514] Processing DeleteTablet for tablet ae56d18c629a4e5eb50911c89de4227c with delete_type TABLET_DATA_TOMBSTONED (TS 05f4fac41c4f4959b9b316cb112d0ada not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:52130
I20240502 21:27:02.458366 5265 tablet_replica.cc:330] T ae56d18c629a4e5eb50911c89de4227c P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:27:02.458568 5265 raft_consensus.cc:2229] T ae56d18c629a4e5eb50911c89de4227c P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:02.458766 5265 raft_consensus.cc:2258] T ae56d18c629a4e5eb50911c89de4227c P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:02.459179 31382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ae56d18c629a4e5eb50911c89de4227c with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:02.461150 5265 ts_tablet_manager.cc:1875] T ae56d18c629a4e5eb50911c89de4227c P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:02.462579 31647 tablet_service.cc:1467] Processing CreateTablet for tablet 925797977edf495ca09aa25988d14fac (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=657b11a3d2954c91a84f226459f11a5b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:02.462955 31647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 925797977edf495ca09aa25988d14fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:02.466288 2076 tablet_service.cc:1467] Processing CreateTablet for tablet 1df21ee31299490aa8bb95dbd9cfd4d0 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=657b11a3d2954c91a84f226459f11a5b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:02.466400 2077 tablet_service.cc:1467] Processing CreateTablet for tablet 565f30d5539c4b7ea956dc49d0f9a55e (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=657b11a3d2954c91a84f226459f11a5b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:02.466738 2076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1df21ee31299490aa8bb95dbd9cfd4d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:02.467052 31775 tablet_service.cc:1467] Processing CreateTablet for tablet 1df21ee31299490aa8bb95dbd9cfd4d0 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=657b11a3d2954c91a84f226459f11a5b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:02.467518 31775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1df21ee31299490aa8bb95dbd9cfd4d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:02.473479 2077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 565f30d5539c4b7ea956dc49d0f9a55e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:02.473517 2210 tablet_service.cc:1467] Processing CreateTablet for tablet 565f30d5539c4b7ea956dc49d0f9a55e (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=657b11a3d2954c91a84f226459f11a5b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:02.473883 2210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 565f30d5539c4b7ea956dc49d0f9a55e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:02.474094 2208 tablet_service.cc:1467] Processing CreateTablet for tablet 925797977edf495ca09aa25988d14fac (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=657b11a3d2954c91a84f226459f11a5b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:02.474496 2208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 925797977edf495ca09aa25988d14fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:02.476716 1946 tablet_service.cc:1467] Processing CreateTablet for tablet 925797977edf495ca09aa25988d14fac (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=657b11a3d2954c91a84f226459f11a5b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:02.476845 1944 tablet_service.cc:1467] Processing CreateTablet for tablet 1df21ee31299490aa8bb95dbd9cfd4d0 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=657b11a3d2954c91a84f226459f11a5b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:02.477098 1946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 925797977edf495ca09aa25988d14fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:02.477270 1944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1df21ee31299490aa8bb95dbd9cfd4d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:02.478850 5265 ts_tablet_manager.cc:1888] T ae56d18c629a4e5eb50911c89de4227c P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:27:02.478978 5265 log.cc:1196] T ae56d18c629a4e5eb50911c89de4227c P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/ae56d18c629a4e5eb50911c89de4227c
I20240502 21:27:02.479791 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet ae56d18c629a4e5eb50911c89de4227c (table rebalance_test_table_5 [id=9e294586148643e080e91357bcbffb17]) successfully deleted
I20240502 21:27:02.480760 1945 tablet_service.cc:1467] Processing CreateTablet for tablet 565f30d5539c4b7ea956dc49d0f9a55e (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=657b11a3d2954c91a84f226459f11a5b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:02.481087 1945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 565f30d5539c4b7ea956dc49d0f9a55e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:02.488906 5316 tablet_bootstrap.cc:492] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:27:02.490808 5316 tablet_bootstrap.cc:654] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:02.492079 5318 tablet_bootstrap.cc:492] T 925797977edf495ca09aa25988d14fac P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:27:02.493885 5318 tablet_bootstrap.cc:654] T 925797977edf495ca09aa25988d14fac P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:02.494510 31647 tablet_service.cc:1514] Processing DeleteTablet for tablet ae56d18c629a4e5eb50911c89de4227c with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 8 (current committed config index is 9)) from {username='slave'} at 127.0.0.1:52130
I20240502 21:27:02.499892 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet ae56d18c629a4e5eb50911c89de4227c (table rebalance_test_table_5 [id=9e294586148643e080e91357bcbffb17]) successfully deleted
I20240502 21:27:02.500254 5318 tablet_bootstrap.cc:492] T 925797977edf495ca09aa25988d14fac P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:27:02.500566 5318 ts_tablet_manager.cc:1367] T 925797977edf495ca09aa25988d14fac P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240502 21:27:02.501556 5318 raft_consensus.cc:348] T 925797977edf495ca09aa25988d14fac P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:02.503180 5322 tablet_bootstrap.cc:492] T 925797977edf495ca09aa25988d14fac P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:27:02.503968 5316 tablet_bootstrap.cc:492] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:27:02.504161 5316 ts_tablet_manager.cc:1367] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.004s
I20240502 21:27:02.504312 31393 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:02.504827 31393 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:02.505841 5316 raft_consensus.cc:348] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:02.506146 5316 raft_consensus.cc:374] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:02.506278 5316 raft_consensus.cc:729] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:27:02.506561 5316 consensus_queue.cc:260] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:02.506755 5320 tablet_bootstrap.cc:492] T 925797977edf495ca09aa25988d14fac P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
W20240502 21:27:02.508824 31860 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.226.134:43753, user_credentials={real_user=slave}} blocked reactor thread for 57112us
I20240502 21:27:02.510092 5316 ts_tablet_manager.cc:1398] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 21:27:02.510355 5316 tablet_bootstrap.cc:492] T 565f30d5539c4b7ea956dc49d0f9a55e P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:27:02.511696 31393 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet c490011176154e38b012cda0bbd695b2
I20240502 21:27:02.511941 5318 raft_consensus.cc:374] T 925797977edf495ca09aa25988d14fac P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:02.512128 5318 raft_consensus.cc:729] T 925797977edf495ca09aa25988d14fac P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:27:02.512720 5318 consensus_queue.cc:260] T 925797977edf495ca09aa25988d14fac P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:02.512856 1946 tablet_service.cc:1514] Processing DeleteTablet for tablet c490011176154e38b012cda0bbd695b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:02 UTC) from {username='slave'} at 127.0.0.1:33002
I20240502 21:27:02.513671 1814 tablet_service.cc:1514] Processing DeleteTablet for tablet c490011176154e38b012cda0bbd695b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:02 UTC) from {username='slave'} at 127.0.0.1:47510
I20240502 21:27:02.515682 5215 raft_consensus.cc:979] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db: : Instructing follower 1ade0354c44c4c37ae480c7a94b11e1b to start an election
I20240502 21:27:02.515842 5215 raft_consensus.cc:1067] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Signalling peer 1ade0354c44c4c37ae480c7a94b11e1b to start an election
I20240502 21:27:02.516155 5318 ts_tablet_manager.cc:1398] T 925797977edf495ca09aa25988d14fac P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240502 21:27:02.516368 5316 tablet_bootstrap.cc:654] T 565f30d5539c4b7ea956dc49d0f9a55e P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:02.516939 2094 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2c737c49a32b4f3ea5ef735fa254d48e"
dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
from {username='slave'} at 127.24.226.133:57191
I20240502 21:27:02.517205 2094 raft_consensus.cc:482] T 2c737c49a32b4f3ea5ef735fa254d48e P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:02.517314 2094 raft_consensus.cc:3037] T 2c737c49a32b4f3ea5ef735fa254d48e P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:02.518977 5322 tablet_bootstrap.cc:654] T 925797977edf495ca09aa25988d14fac P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:02.521449 5317 tablet_bootstrap.cc:492] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:27:02.522010 2094 raft_consensus.cc:504] T 2c737c49a32b4f3ea5ef735fa254d48e P 1ade0354c44c4c37ae480c7a94b11e1b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } }
I20240502 21:27:02.523411 5317 tablet_bootstrap.cc:654] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:02.524686 1833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c737c49a32b4f3ea5ef735fa254d48e" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:27:02.524947 1833 raft_consensus.cc:3032] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:02.525044 1833 raft_consensus.cc:729] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Running, Role: LEADER
I20240502 21:27:02.525301 5320 tablet_bootstrap.cc:654] T 925797977edf495ca09aa25988d14fac P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:02.525386 1833 consensus_queue.cc:260] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } }
I20240502 21:27:02.525931 1833 raft_consensus.cc:3037] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:02.528705 1833 raft_consensus.cc:2454] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 2.
I20240502 21:27:02.530433 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c737c49a32b4f3ea5ef735fa254d48e" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:27:02.530648 31797 raft_consensus.cc:3037] T 2c737c49a32b4f3ea5ef735fa254d48e P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:02.531160 5325 tablet_replica.cc:330] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db: stopping tablet replica
I20240502 21:27:02.532814 5325 raft_consensus.cc:2229] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:02.533391 31797 raft_consensus.cc:2454] T 2c737c49a32b4f3ea5ef735fa254d48e P 08d7a0680d244b55b93d17f9ab57b84c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 2.
I20240502 21:27:02.534114 5322 tablet_bootstrap.cc:492] T 925797977edf495ca09aa25988d14fac P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:27:02.534386 5322 ts_tablet_manager.cc:1367] T 925797977edf495ca09aa25988d14fac P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.000s
I20240502 21:27:02.534958 2032 leader_election.cc:302] T 2c737c49a32b4f3ea5ef735fa254d48e P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 1ade0354c44c4c37ae480c7a94b11e1b, 1e04157b98dd4fad98858c290b73c3db; no voters:
I20240502 21:27:02.535240 4904 raft_consensus.cc:2781] T 2c737c49a32b4f3ea5ef735fa254d48e P 1ade0354c44c4c37ae480c7a94b11e1b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:02.535305 5325 raft_consensus.cc:2258] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:02.535431 4904 raft_consensus.cc:686] T 2c737c49a32b4f3ea5ef735fa254d48e P 1ade0354c44c4c37ae480c7a94b11e1b [term 2 LEADER]: Becoming Leader. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Running, Role: LEADER
I20240502 21:27:02.535765 4904 consensus_queue.cc:237] T 2c737c49a32b4f3ea5ef735fa254d48e P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } }
I20240502 21:27:02.535831 5322 raft_consensus.cc:348] T 925797977edf495ca09aa25988d14fac P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:02.536144 5322 raft_consensus.cc:374] T 925797977edf495ca09aa25988d14fac P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:02.536285 5322 raft_consensus.cc:729] T 925797977edf495ca09aa25988d14fac P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:27:02.536629 5322 consensus_queue.cc:260] T 925797977edf495ca09aa25988d14fac P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:02.538383 5322 ts_tablet_manager.cc:1398] T 925797977edf495ca09aa25988d14fac P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:02.539106 5322 tablet_bootstrap.cc:492] T 565f30d5539c4b7ea956dc49d0f9a55e P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:27:02.540269 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c737c49a32b4f3ea5ef735fa254d48e" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:27:02.540470 31665 raft_consensus.cc:3037] T 2c737c49a32b4f3ea5ef735fa254d48e P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:02.541193 5322 tablet_bootstrap.cc:654] T 565f30d5539c4b7ea956dc49d0f9a55e P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:02.542115 5325 ts_tablet_manager.cc:1875] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:02.542925 31665 raft_consensus.cc:2454] T 2c737c49a32b4f3ea5ef735fa254d48e P 05f4fac41c4f4959b9b316cb112d0ada [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 2.
I20240502 21:27:02.545557 2094 leader_election.cc:288] T 2c737c49a32b4f3ea5ef735fa254d48e P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 election: Requested vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:27:02.548208 5322 tablet_bootstrap.cc:492] T 565f30d5539c4b7ea956dc49d0f9a55e P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:27:02.548403 5322 ts_tablet_manager.cc:1367] T 565f30d5539c4b7ea956dc49d0f9a55e P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20240502 21:27:02.550339 2208 tablet_service.cc:1514] Processing DeleteTablet for tablet c490011176154e38b012cda0bbd695b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:02 UTC) from {username='slave'} at 127.0.0.1:36984
I20240502 21:27:02.550643 5284 tablet_replica.cc:330] T c490011176154e38b012cda0bbd695b2 P d1406e92aec84ac585304e57bd37a40d: stopping tablet replica
I20240502 21:27:02.550841 5284 raft_consensus.cc:2229] T c490011176154e38b012cda0bbd695b2 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:02.551034 5284 raft_consensus.cc:2258] T c490011176154e38b012cda0bbd695b2 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:02.552088 5284 ts_tablet_manager.cc:1875] T c490011176154e38b012cda0bbd695b2 P d1406e92aec84ac585304e57bd37a40d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:02.552227 5324 tablet_replica.cc:330] T c490011176154e38b012cda0bbd695b2 P 9dec645b473f40578e726f3e50e219bf: stopping tablet replica
I20240502 21:27:02.552407 5324 raft_consensus.cc:2229] T c490011176154e38b012cda0bbd695b2 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:02.552623 5324 raft_consensus.cc:2258] T c490011176154e38b012cda0bbd695b2 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:02.554461 31392 catalog_manager.cc:5484] T 2c737c49a32b4f3ea5ef735fa254d48e P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: term changed from 1 to 2, leader changed from 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) to 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135). New cstate: current_term: 2 leader_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:02.552868 5320 tablet_bootstrap.cc:492] T 925797977edf495ca09aa25988d14fac P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:27:02.555426 5320 ts_tablet_manager.cc:1367] T 925797977edf495ca09aa25988d14fac P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.049s user 0.005s sys 0.000s
I20240502 21:27:02.556238 5320 raft_consensus.cc:348] T 925797977edf495ca09aa25988d14fac P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:02.558025 5317 tablet_bootstrap.cc:492] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:27:02.558218 5317 ts_tablet_manager.cc:1367] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.037s user 0.007s sys 0.021s
I20240502 21:27:02.560266 5320 raft_consensus.cc:374] T 925797977edf495ca09aa25988d14fac P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:02.560261 5317 raft_consensus.cc:348] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:02.560400 5320 raft_consensus.cc:729] T 925797977edf495ca09aa25988d14fac P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:27:02.560178 5324 ts_tablet_manager.cc:1875] T c490011176154e38b012cda0bbd695b2 P 9dec645b473f40578e726f3e50e219bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:02.560570 5317 raft_consensus.cc:374] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:02.560645 5317 raft_consensus.cc:729] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:27:02.561954 5284 ts_tablet_manager.cc:1888] T c490011176154e38b012cda0bbd695b2 P d1406e92aec84ac585304e57bd37a40d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:02.561826 5317 consensus_queue.cc:260] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:02.562135 5284 log.cc:1196] T c490011176154e38b012cda0bbd695b2 P d1406e92aec84ac585304e57bd37a40d: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal/wals/c490011176154e38b012cda0bbd695b2
I20240502 21:27:02.562577 5284 ts_tablet_manager.cc:1909] T c490011176154e38b012cda0bbd695b2 P d1406e92aec84ac585304e57bd37a40d: Deleting consensus metadata
I20240502 21:27:02.563551 31381 catalog_manager.cc:4830] TS d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): tablet c490011176154e38b012cda0bbd695b2 (table rebalancer_dynamic_table_new_name [id=0cd367f3616542169f8d3fc53d47e806]) successfully deleted
I20240502 21:27:02.551034 5322 raft_consensus.cc:348] T 565f30d5539c4b7ea956dc49d0f9a55e P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:02.564240 5322 raft_consensus.cc:374] T 565f30d5539c4b7ea956dc49d0f9a55e P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:02.564355 5322 raft_consensus.cc:729] T 565f30d5539c4b7ea956dc49d0f9a55e P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:27:02.564544 5320 consensus_queue.cc:260] T 925797977edf495ca09aa25988d14fac P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:02.565241 5320 ts_tablet_manager.cc:1398] T 925797977edf495ca09aa25988d14fac P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240502 21:27:02.565411 5316 tablet_bootstrap.cc:492] T 565f30d5539c4b7ea956dc49d0f9a55e P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:27:02.565642 5316 ts_tablet_manager.cc:1367] T 565f30d5539c4b7ea956dc49d0f9a55e P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.055s user 0.005s sys 0.009s
I20240502 21:27:02.565754 5320 tablet_bootstrap.cc:492] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:27:02.566241 5317 ts_tablet_manager.cc:1398] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20240502 21:27:02.566394 5316 raft_consensus.cc:348] T 565f30d5539c4b7ea956dc49d0f9a55e P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:02.566617 5316 raft_consensus.cc:374] T 565f30d5539c4b7ea956dc49d0f9a55e P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:02.566710 5316 raft_consensus.cc:729] T 565f30d5539c4b7ea956dc49d0f9a55e P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:27:02.566974 5316 consensus_queue.cc:260] T 565f30d5539c4b7ea956dc49d0f9a55e P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:02.567648 5316 ts_tablet_manager.cc:1398] T 565f30d5539c4b7ea956dc49d0f9a55e P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:27:02.564718 5322 consensus_queue.cc:260] T 565f30d5539c4b7ea956dc49d0f9a55e P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:02.568343 5325 ts_tablet_manager.cc:1888] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:02.569753 5322 ts_tablet_manager.cc:1398] T 565f30d5539c4b7ea956dc49d0f9a55e P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.020s user 0.000s sys 0.002s
I20240502 21:27:02.570559 5320 tablet_bootstrap.cc:654] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:02.572398 5325 log.cc:1196] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal/wals/c490011176154e38b012cda0bbd695b2
I20240502 21:27:02.573444 5325 ts_tablet_manager.cc:1909] T c490011176154e38b012cda0bbd695b2 P 1e04157b98dd4fad98858c290b73c3db: Deleting consensus metadata
I20240502 21:27:02.576285 5324 ts_tablet_manager.cc:1888] T c490011176154e38b012cda0bbd695b2 P 9dec645b473f40578e726f3e50e219bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:02.576412 5324 log.cc:1196] T c490011176154e38b012cda0bbd695b2 P 9dec645b473f40578e726f3e50e219bf: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal/wals/c490011176154e38b012cda0bbd695b2
I20240502 21:27:02.578073 4559 raft_consensus.cc:482] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:02.578202 4559 raft_consensus.cc:504] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:02.578526 31381 catalog_manager.cc:4830] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): tablet c490011176154e38b012cda0bbd695b2 (table rebalancer_dynamic_table_new_name [id=0cd367f3616542169f8d3fc53d47e806]) successfully deleted
I20240502 21:27:02.578609 5324 ts_tablet_manager.cc:1909] T c490011176154e38b012cda0bbd695b2 P 9dec645b473f40578e726f3e50e219bf: Deleting consensus metadata
I20240502 21:27:02.578693 5320 tablet_bootstrap.cc:492] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:27:02.578819 4559 leader_election.cc:288] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:27:02.578917 5320 ts_tablet_manager.cc:1367] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240502 21:27:02.579555 2094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df21ee31299490aa8bb95dbd9cfd4d0" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:27:02.579778 2094 raft_consensus.cc:2454] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:27:02.579699 5320 raft_consensus.cc:348] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:02.579893 5320 raft_consensus.cc:374] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:02.579958 5320 raft_consensus.cc:729] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:27:02.580210 31732 leader_election.cc:302] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 1ade0354c44c4c37ae480c7a94b11e1b; no voters:
I20240502 21:27:02.580250 5320 consensus_queue.cc:260] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:02.580623 4559 raft_consensus.cc:2781] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:02.580713 4559 raft_consensus.cc:482] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:02.580799 4559 raft_consensus.cc:3037] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:02.580969 5320 ts_tablet_manager.cc:1398] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:02.581243 5320 tablet_bootstrap.cc:492] T 565f30d5539c4b7ea956dc49d0f9a55e P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:27:02.581470 1964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df21ee31299490aa8bb95dbd9cfd4d0" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:27:02.581625 1964 raft_consensus.cc:2454] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:27:02.582720 5320 tablet_bootstrap.cc:654] T 565f30d5539c4b7ea956dc49d0f9a55e P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:02.583209 4559 raft_consensus.cc:504] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:02.583858 4559 leader_election.cc:288] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Requested vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:27:02.584064 31379 catalog_manager.cc:4830] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): tablet c490011176154e38b012cda0bbd695b2 (table rebalancer_dynamic_table_new_name [id=0cd367f3616542169f8d3fc53d47e806]) successfully deleted
I20240502 21:27:02.584932 2094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df21ee31299490aa8bb95dbd9cfd4d0" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:27:02.585186 2094 raft_consensus.cc:3037] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:02.589496 1964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df21ee31299490aa8bb95dbd9cfd4d0" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf"
I20240502 21:27:02.589715 1964 raft_consensus.cc:3037] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:02.588423 2094 raft_consensus.cc:2454] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:27:02.591914 1964 raft_consensus.cc:2454] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:27:02.592311 31730 leader_election.cc:302] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:27:02.592669 4559 raft_consensus.cc:2781] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:02.593245 4559 raft_consensus.cc:686] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Becoming Leader. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Running, Role: LEADER
I20240502 21:27:02.593565 4559 consensus_queue.cc:237] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:02.597927 31392 catalog_manager.cc:5484] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: term changed from 0 to 1, leader changed from <none> to 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131). New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:02.601999 5320 tablet_bootstrap.cc:492] T 565f30d5539c4b7ea956dc49d0f9a55e P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:27:02.602193 5320 ts_tablet_manager.cc:1367] T 565f30d5539c4b7ea956dc49d0f9a55e P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.018s
I20240502 21:27:02.602995 5320 raft_consensus.cc:348] T 565f30d5539c4b7ea956dc49d0f9a55e P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:02.603180 5320 raft_consensus.cc:374] T 565f30d5539c4b7ea956dc49d0f9a55e P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:02.603266 5320 raft_consensus.cc:729] T 565f30d5539c4b7ea956dc49d0f9a55e P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:27:02.603459 5320 consensus_queue.cc:260] T 565f30d5539c4b7ea956dc49d0f9a55e P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:02.603968 5320 ts_tablet_manager.cc:1398] T 565f30d5539c4b7ea956dc49d0f9a55e P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:02.624043 31392 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:02.625553 3818 raft_consensus.cc:482] T 565f30d5539c4b7ea956dc49d0f9a55e P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:02.625731 3818 raft_consensus.cc:504] T 565f30d5539c4b7ea956dc49d0f9a55e P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:02.626304 3818 leader_election.cc:288] T 565f30d5539c4b7ea956dc49d0f9a55e P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:27:02.626646 2094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "565f30d5539c4b7ea956dc49d0f9a55e" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:27:02.626828 2094 raft_consensus.cc:2454] T 565f30d5539c4b7ea956dc49d0f9a55e P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 0.
I20240502 21:27:02.626801 2230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "565f30d5539c4b7ea956dc49d0f9a55e" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:27:02.626988 2230 raft_consensus.cc:2454] T 565f30d5539c4b7ea956dc49d0f9a55e P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 0.
I20240502 21:27:02.627579 1833 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3e6657723fe84e749dea768de44638e7"
dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59770
I20240502 21:27:02.627764 1833 raft_consensus.cc:595] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Received request to transfer leadership
I20240502 21:27:02.628031 1901 leader_election.cc:302] T 565f30d5539c4b7ea956dc49d0f9a55e P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dec645b473f40578e726f3e50e219bf, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:27:02.628481 3818 raft_consensus.cc:2781] T 565f30d5539c4b7ea956dc49d0f9a55e P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:02.628582 3818 raft_consensus.cc:482] T 565f30d5539c4b7ea956dc49d0f9a55e P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:02.628674 3818 raft_consensus.cc:3037] T 565f30d5539c4b7ea956dc49d0f9a55e P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:02.630976 3818 raft_consensus.cc:504] T 565f30d5539c4b7ea956dc49d0f9a55e P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:02.631767 2094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "565f30d5539c4b7ea956dc49d0f9a55e" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:27:02.631963 2094 raft_consensus.cc:3037] T 565f30d5539c4b7ea956dc49d0f9a55e P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:02.631980 3818 leader_election.cc:288] T 565f30d5539c4b7ea956dc49d0f9a55e P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 election: Requested vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:27:02.634065 31906 tablet_service.cc:1467] Processing CreateTablet for tablet 654b4e1fe654455b855b7dae1203badf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3880230f55a6456b8789cd24843fd807]), partition=
I20240502 21:27:02.634429 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 654b4e1fe654455b855b7dae1203badf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:02.634444 2094 raft_consensus.cc:2454] T 565f30d5539c4b7ea956dc49d0f9a55e P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 1.
I20240502 21:27:02.634871 1901 leader_election.cc:302] T 565f30d5539c4b7ea956dc49d0f9a55e P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:27:02.635149 3818 raft_consensus.cc:2781] T 565f30d5539c4b7ea956dc49d0f9a55e P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:02.635233 1946 tablet_service.cc:1467] Processing CreateTablet for tablet 654b4e1fe654455b855b7dae1203badf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3880230f55a6456b8789cd24843fd807]), partition=
I20240502 21:27:02.635237 31647 tablet_service.cc:1467] Processing CreateTablet for tablet 654b4e1fe654455b855b7dae1203badf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3880230f55a6456b8789cd24843fd807]), partition=
I20240502 21:27:02.635562 1946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 654b4e1fe654455b855b7dae1203badf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:02.635582 31647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 654b4e1fe654455b855b7dae1203badf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:02.635622 3818 raft_consensus.cc:686] T 565f30d5539c4b7ea956dc49d0f9a55e P 9dec645b473f40578e726f3e50e219bf [term 1 LEADER]: Becoming Leader. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Running, Role: LEADER
I20240502 21:27:02.636123 3818 consensus_queue.cc:237] T 565f30d5539c4b7ea956dc49d0f9a55e P 9dec645b473f40578e726f3e50e219bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:02.640906 5318 tablet_bootstrap.cc:492] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:27:02.642452 5318 tablet_bootstrap.cc:654] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:02.642529 2230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "565f30d5539c4b7ea956dc49d0f9a55e" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:27:02.642732 2230 raft_consensus.cc:3037] T 565f30d5539c4b7ea956dc49d0f9a55e P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:02.645927 2230 raft_consensus.cc:2454] T 565f30d5539c4b7ea956dc49d0f9a55e P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 1.
I20240502 21:27:02.654244 5318 tablet_bootstrap.cc:492] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:27:02.654439 5318 ts_tablet_manager.cc:1367] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20240502 21:27:02.655170 5318 raft_consensus.cc:348] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:27:02.655459 5318 raft_consensus.cc:374] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:02.655582 5318 raft_consensus.cc:729] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:27:02.655869 5318 consensus_queue.cc:260] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:27:02.657613 5320 tablet_bootstrap.cc:492] T 654b4e1fe654455b855b7dae1203badf P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:27:02.659312 5320 tablet_bootstrap.cc:654] T 654b4e1fe654455b855b7dae1203badf P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:02.662811 5318 ts_tablet_manager.cc:1398] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240502 21:27:02.667343 31392 catalog_manager.cc:5484] T 565f30d5539c4b7ea956dc49d0f9a55e P 9dec645b473f40578e726f3e50e219bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dec645b473f40578e726f3e50e219bf (127.24.226.134). New cstate: current_term: 1 leader_uuid: "9dec645b473f40578e726f3e50e219bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:02.675518 5346 tablet_bootstrap.cc:492] T 654b4e1fe654455b855b7dae1203badf P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:27:02.679029 5320 tablet_bootstrap.cc:492] T 654b4e1fe654455b855b7dae1203badf P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:27:02.679291 5320 ts_tablet_manager.cc:1367] T 654b4e1fe654455b855b7dae1203badf P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.008s
I20240502 21:27:02.680789 5346 tablet_bootstrap.cc:654] T 654b4e1fe654455b855b7dae1203badf P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:02.684558 5320 raft_consensus.cc:348] T 654b4e1fe654455b855b7dae1203badf P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:27:02.684820 5320 raft_consensus.cc:374] T 654b4e1fe654455b855b7dae1203badf P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:02.684907 5320 raft_consensus.cc:729] T 654b4e1fe654455b855b7dae1203badf P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:27:02.685148 5320 consensus_queue.cc:260] T 654b4e1fe654455b855b7dae1203badf P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:27:02.691231 5320 ts_tablet_manager.cc:1398] T 654b4e1fe654455b855b7dae1203badf P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240502 21:27:02.693742 5346 tablet_bootstrap.cc:492] T 654b4e1fe654455b855b7dae1203badf P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:27:02.693908 5346 ts_tablet_manager.cc:1367] T 654b4e1fe654455b855b7dae1203badf P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.006s
I20240502 21:27:02.695039 5346 raft_consensus.cc:348] T 654b4e1fe654455b855b7dae1203badf P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:27:02.695340 5346 raft_consensus.cc:374] T 654b4e1fe654455b855b7dae1203badf P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:02.695437 5346 raft_consensus.cc:729] T 654b4e1fe654455b855b7dae1203badf P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:27:02.695753 5346 consensus_queue.cc:260] T 654b4e1fe654455b855b7dae1203badf P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:27:02.696455 5346 ts_tablet_manager.cc:1398] T 654b4e1fe654455b855b7dae1203badf P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:02.721289 31533 consensus_queue.cc:237] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:27:02.723342 2094 raft_consensus.cc:1261] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 21:27:02.723371 31469 consensus_peers.cc:485] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5bc6c7211d5f4f73b2a050b9ec78c6b1. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:02.723498 31924 raft_consensus.cc:1261] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:02.723932 5090 consensus_queue.cc:1035] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:02.724359 5089 consensus_queue.cc:1035] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:02.727449 5090 raft_consensus.cc:2932] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:27:02.728238 2094 raft_consensus.cc:2932] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:27:02.734192 31386 catalog_manager.cc:5484] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: config changed from index 4 to 5, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:02.742635 5201 raft_consensus.cc:482] T 925797977edf495ca09aa25988d14fac P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:02.742790 5201 raft_consensus.cc:504] T 925797977edf495ca09aa25988d14fac P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:02.743490 5201 leader_election.cc:288] T 925797977edf495ca09aa25988d14fac P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:27:02.744056 2230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925797977edf495ca09aa25988d14fac" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:27:02.744019 1965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925797977edf495ca09aa25988d14fac" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:27:02.744287 2230 raft_consensus.cc:2454] T 925797977edf495ca09aa25988d14fac P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:27:02.744287 1965 raft_consensus.cc:2454] T 925797977edf495ca09aa25988d14fac P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:27:02.744745 31602 leader_election.cc:302] T 925797977edf495ca09aa25988d14fac P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:27:02.745792 5201 raft_consensus.cc:2781] T 925797977edf495ca09aa25988d14fac P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:02.745906 5201 raft_consensus.cc:482] T 925797977edf495ca09aa25988d14fac P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:02.745983 5201 raft_consensus.cc:3037] T 925797977edf495ca09aa25988d14fac P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:02.745507 31924 raft_consensus.cc:2932] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:27:02.748114 5201 raft_consensus.cc:504] T 925797977edf495ca09aa25988d14fac P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:02.749704 1965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925797977edf495ca09aa25988d14fac" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf"
I20240502 21:27:02.749917 1965 raft_consensus.cc:3037] T 925797977edf495ca09aa25988d14fac P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:02.752038 1965 raft_consensus.cc:2454] T 925797977edf495ca09aa25988d14fac P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:27:02.752699 31600 leader_election.cc:302] T 925797977edf495ca09aa25988d14fac P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:27:02.753008 5069 raft_consensus.cc:2781] T 925797977edf495ca09aa25988d14fac P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:02.754204 5201 leader_election.cc:288] T 925797977edf495ca09aa25988d14fac P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Requested vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:27:02.754498 2224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925797977edf495ca09aa25988d14fac" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:27:02.754762 2224 raft_consensus.cc:3037] T 925797977edf495ca09aa25988d14fac P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:02.757764 2224 raft_consensus.cc:2454] T 925797977edf495ca09aa25988d14fac P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:27:02.764998 5069 raft_consensus.cc:686] T 925797977edf495ca09aa25988d14fac P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Becoming Leader. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Running, Role: LEADER
I20240502 21:27:02.765345 5069 consensus_queue.cc:237] T 925797977edf495ca09aa25988d14fac P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:02.769632 31386 catalog_manager.cc:5484] T 925797977edf495ca09aa25988d14fac P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130). New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:02.788267 31665 consensus_queue.cc:237] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } }
I20240502 21:27:02.790788 31797 raft_consensus.cc:1261] T f61e34e65429408e83a13c7bc743e78a P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:27:02.790942 31602 consensus_peers.cc:485] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): Couldn't send request to peer 1ade0354c44c4c37ae480c7a94b11e1b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f61e34e65429408e83a13c7bc743e78a. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:02.791658 31924 raft_consensus.cc:1261] T f61e34e65429408e83a13c7bc743e78a P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:02.791677 5367 consensus_queue.cc:1035] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:02.792098 5201 consensus_queue.cc:1035] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:02.805379 5201 raft_consensus.cc:2932] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } } }
I20240502 21:27:02.810685 31924 raft_consensus.cc:2932] T f61e34e65429408e83a13c7bc743e78a P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } } }
I20240502 21:27:02.815413 31386 catalog_manager.cc:5484] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:02.816191 31797 raft_consensus.cc:2932] T f61e34e65429408e83a13c7bc743e78a P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } } }
I20240502 21:27:02.837086 31924 consensus_queue.cc:237] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } }
I20240502 21:27:02.839403 2224 raft_consensus.cc:1261] T 1f743c7345184feaa50871dc49a1f136 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:27:02.839567 31862 consensus_peers.cc:485] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 -> Peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): Couldn't send request to peer 08d7a0680d244b55b93d17f9ab57b84c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1f743c7345184feaa50871dc49a1f136. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:02.839747 31533 raft_consensus.cc:1261] T 1f743c7345184feaa50871dc49a1f136 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:02.839890 5202 consensus_queue.cc:1035] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:02.840220 5315 consensus_queue.cc:1035] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:02.842517 5303 raft_consensus.cc:2932] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } } }
I20240502 21:27:02.843108 2224 raft_consensus.cc:2932] T 1f743c7345184feaa50871dc49a1f136 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } } }
I20240502 21:27:02.849496 31393 catalog_manager.cc:5484] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: config changed from index -1 to 2, NON_VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) added. New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:02.866691 31533 raft_consensus.cc:2932] T 1f743c7345184feaa50871dc49a1f136 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } } }
I20240502 21:27:02.886185 31533 consensus_queue.cc:237] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } }
I20240502 21:27:02.888154 2094 raft_consensus.cc:1261] T 4bc968ae4dcc4636a49257b6d5824bee P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:02.888314 1965 raft_consensus.cc:1261] T 4bc968ae4dcc4636a49257b6d5824bee P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:27:02.888350 31472 consensus_peers.cc:485] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f -> Peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): Couldn't send request to peer 05f4fac41c4f4959b9b316cb112d0ada. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4bc968ae4dcc4636a49257b6d5824bee. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:02.888813 5090 consensus_queue.cc:1035] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:02.889081 4942 consensus_queue.cc:1035] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:02.893263 5090 raft_consensus.cc:2932] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } } }
I20240502 21:27:02.893788 2094 raft_consensus.cc:2932] T 4bc968ae4dcc4636a49257b6d5824bee P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } } }
I20240502 21:27:02.899652 5367 raft_consensus.cc:482] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:02.899791 5367 raft_consensus.cc:504] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:27:02.900146 31393 catalog_manager.cc:5484] T 4bc968ae4dcc4636a49257b6d5824bee P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } } }
I20240502 21:27:02.902132 31924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654b4e1fe654455b855b7dae1203badf" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:27:02.902315 31924 raft_consensus.cc:2454] T 654b4e1fe654455b855b7dae1203badf P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:27:02.902741 31603 leader_election.cc:302] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:27:02.902992 5069 raft_consensus.cc:2781] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:02.903057 5069 raft_consensus.cc:482] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:02.903138 5069 raft_consensus.cc:3037] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:02.903090 1964 raft_consensus.cc:2932] T 4bc968ae4dcc4636a49257b6d5824bee P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } } }
I20240502 21:27:02.900388 5367 leader_election.cc:288] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:27:02.905562 5069 raft_consensus.cc:504] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:27:02.906759 1962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654b4e1fe654455b855b7dae1203badf" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf"
I20240502 21:27:02.906950 1962 raft_consensus.cc:3037] T 654b4e1fe654455b855b7dae1203badf P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:02.908758 1962 raft_consensus.cc:2454] T 654b4e1fe654455b855b7dae1203badf P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:27:02.909181 1965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654b4e1fe654455b855b7dae1203badf" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:27:02.909372 1965 raft_consensus.cc:2362] T 654b4e1fe654455b855b7dae1203badf P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1. Re-sending same reply.
I20240502 21:27:02.910104 31600 leader_election.cc:302] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:27:02.911051 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654b4e1fe654455b855b7dae1203badf" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:27:02.911279 31925 raft_consensus.cc:3037] T 654b4e1fe654455b855b7dae1203badf P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:02.911401 5367 raft_consensus.cc:2781] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:02.911545 5367 raft_consensus.cc:686] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Becoming Leader. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Running, Role: LEADER
I20240502 21:27:02.911823 5367 consensus_queue.cc:237] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:27:02.913908 5069 leader_election.cc:288] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Requested vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:27:02.914453 31925 raft_consensus.cc:2454] T 654b4e1fe654455b855b7dae1203badf P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:27:02.921327 31392 catalog_manager.cc:5484] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130). New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:02.930682 2224 consensus_queue.cc:237] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } }
I20240502 21:27:02.933192 2094 raft_consensus.cc:1261] T 883c729aa26f47178f40a17ad3b7e590 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:27:02.933260 2164 consensus_peers.cc:485] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d -> Peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): Couldn't send request to peer 2d50f33557944bb480ea32199d9ee4f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 883c729aa26f47178f40a17ad3b7e590. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:02.933275 1832 raft_consensus.cc:1261] T 883c729aa26f47178f40a17ad3b7e590 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:02.933626 5400 consensus_queue.cc:1035] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:02.933876 5156 consensus_queue.cc:1035] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:02.936281 5400 raft_consensus.cc:2932] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } } }
I20240502 21:27:02.936842 2094 raft_consensus.cc:2932] T 883c729aa26f47178f40a17ad3b7e590 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } } }
I20240502 21:27:02.938944 1832 raft_consensus.cc:2932] T 883c729aa26f47178f40a17ad3b7e590 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } } }
I20240502 21:27:02.941992 31392 catalog_manager.cc:5484] T 883c729aa26f47178f40a17ad3b7e590 P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: config changed from index -1 to 2, NON_VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) added. New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } } }
I20240502 21:27:02.958988 31533 consensus_queue.cc:237] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } }
I20240502 21:27:02.961004 31925 raft_consensus.cc:1261] T 381ac1d714204cef86c8e1a73e24b863 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:02.961057 31797 raft_consensus.cc:1261] T 381ac1d714204cef86c8e1a73e24b863 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:27:02.961112 31471 consensus_peers.cc:485] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f -> Peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): Couldn't send request to peer 1e04157b98dd4fad98858c290b73c3db. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 381ac1d714204cef86c8e1a73e24b863. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:02.961477 5089 consensus_queue.cc:1035] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:02.961707 4942 consensus_queue.cc:1035] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:02.964151 5089 raft_consensus.cc:2932] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } } }
I20240502 21:27:02.964802 31925 raft_consensus.cc:2932] T 381ac1d714204cef86c8e1a73e24b863 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } } }
I20240502 21:27:02.965294 31797 raft_consensus.cc:2932] T 381ac1d714204cef86c8e1a73e24b863 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } } }
I20240502 21:27:02.970244 31392 catalog_manager.cc:5484] T 381ac1d714204cef86c8e1a73e24b863 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: config changed from index -1 to 4, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } } }
I20240502 21:27:02.982746 5223 raft_consensus.cc:979] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db: : Instructing follower a44f41b0b6644fb6aa29476aed8c964f to start an election
I20240502 21:27:02.982889 5223 raft_consensus.cc:1067] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Signalling peer a44f41b0b6644fb6aa29476aed8c964f to start an election
I20240502 21:27:02.983501 31533 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3e6657723fe84e749dea768de44638e7"
dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
from {username='slave'} at 127.24.226.133:49939
I20240502 21:27:02.983732 31533 raft_consensus.cc:482] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:02.983837 31533 raft_consensus.cc:3037] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:02.984768 31797 consensus_queue.cc:237] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } }
I20240502 21:27:02.986428 31533 raft_consensus.cc:504] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:27:02.987706 1832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e6657723fe84e749dea768de44638e7" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:27:02.987919 1832 raft_consensus.cc:3032] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:02.988027 1832 raft_consensus.cc:729] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Running, Role: LEADER
I20240502 21:27:02.988337 1832 consensus_queue.cc:260] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:27:02.988754 1832 raft_consensus.cc:3037] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:02.991264 1833 raft_consensus.cc:1261] T b8db5de7df864ab983504e96fb738106 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:02.991585 1832 raft_consensus.cc:2454] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 2.
I20240502 21:27:02.991619 1964 raft_consensus.cc:1261] T b8db5de7df864ab983504e96fb738106 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:02.992364 31533 leader_election.cc:288] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 2 election: Requested vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:27:02.992384 4859 consensus_queue.cc:1035] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:02.992856 4859 consensus_queue.cc:1035] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:02.993636 1965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e6657723fe84e749dea768de44638e7" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9dec645b473f40578e726f3e50e219bf"
I20240502 21:27:02.993819 1965 raft_consensus.cc:3037] T 3e6657723fe84e749dea768de44638e7 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:02.995013 4859 raft_consensus.cc:2932] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } } }
I20240502 21:27:02.995553 1832 raft_consensus.cc:2932] T b8db5de7df864ab983504e96fb738106 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } } }
I20240502 21:27:02.996308 1965 raft_consensus.cc:2454] T 3e6657723fe84e749dea768de44638e7 P 9dec645b473f40578e726f3e50e219bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 2.
I20240502 21:27:02.997030 1833 raft_consensus.cc:1261] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [term 2 FOLLOWER]: Refusing update from remote peer 1ade0354c44c4c37ae480c7a94b11e1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20240502 21:27:02.997568 31733 consensus_peers.cc:485] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c -> Peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): Couldn't send request to peer 05f4fac41c4f4959b9b316cb112d0ada. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8db5de7df864ab983504e96fb738106. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:02.998138 5273 consensus_queue.cc:1035] T 2c737c49a32b4f3ea5ef735fa254d48e P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:02.998208 31469 leader_election.cc:302] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, 9dec645b473f40578e726f3e50e219bf, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:27:02.998458 5089 raft_consensus.cc:2781] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:02.998548 5089 raft_consensus.cc:686] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f [term 2 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:27:02.998754 5089 consensus_queue.cc:237] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:27:03.000058 2092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e6657723fe84e749dea768de44638e7" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:27:03.000252 2092 raft_consensus.cc:3037] T 3e6657723fe84e749dea768de44638e7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:03.000938 31797 raft_consensus.cc:1261] T 2c737c49a32b4f3ea5ef735fa254d48e P 08d7a0680d244b55b93d17f9ab57b84c [term 2 FOLLOWER]: Refusing update from remote peer 1ade0354c44c4c37ae480c7a94b11e1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:27:03.003196 2092 raft_consensus.cc:2454] T 3e6657723fe84e749dea768de44638e7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 2.
I20240502 21:27:03.003736 31389 catalog_manager.cc:5484] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:03.007730 4904 consensus_queue.cc:1035] T 2c737c49a32b4f3ea5ef735fa254d48e P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:03.023521 31393 catalog_manager.cc:5484] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 1 to 2, leader changed from 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 2 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:03.033308 31392 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:03.037415 1964 raft_consensus.cc:2932] T b8db5de7df864ab983504e96fb738106 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } } }
I20240502 21:27:03.051116 4859 consensus_queue.cc:1035] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:03.076092 31665 raft_consensus.cc:1261] T 2c737c49a32b4f3ea5ef735fa254d48e P 05f4fac41c4f4959b9b316cb112d0ada [term 2 FOLLOWER]: Refusing update from remote peer 1ade0354c44c4c37ae480c7a94b11e1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:27:03.076730 5273 consensus_queue.cc:1035] T 2c737c49a32b4f3ea5ef735fa254d48e P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:03.098832 4859 consensus_queue.cc:1035] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:03.100605 1965 raft_consensus.cc:1261] T 654b4e1fe654455b855b7dae1203badf P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:03.101150 5367 consensus_queue.cc:1035] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:03.103349 31925 raft_consensus.cc:1261] T 654b4e1fe654455b855b7dae1203badf P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:03.103829 5367 consensus_queue.cc:1035] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:03.154412 31392 catalog_manager.cc:5576] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753) does not have the latest schema for tablet 654b4e1fe654455b855b7dae1203badf (table rebalancer_dynamic_table [id=3880230f55a6456b8789cd24843fd807]). Expected version 1 got 0
I20240502 21:27:03.170773 5201 consensus_queue.cc:1035] T 925797977edf495ca09aa25988d14fac P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:03.175458 5201 consensus_queue.cc:1035] T 925797977edf495ca09aa25988d14fac P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:03.187975 5438 tablet.cc:1722] T 654b4e1fe654455b855b7dae1203badf P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:03.191042 5438 tablet.cc:1620] T 654b4e1fe654455b855b7dae1203badf P 9dec645b473f40578e726f3e50e219bf: MemRowSet was empty: no flush needed.
I20240502 21:27:03.192226 31392 catalog_manager.cc:5576] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753) does not have the latest schema for tablet 654b4e1fe654455b855b7dae1203badf (table rebalancer_dynamic_table [id=3880230f55a6456b8789cd24843fd807]). Expected version 1 got 0
I20240502 21:27:03.200913 5345 consensus_queue.cc:1035] T 565f30d5539c4b7ea956dc49d0f9a55e P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:03.219018 5437 tablet.cc:1722] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:03.219560 5437 tablet.cc:1620] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada: MemRowSet was empty: no flush needed.
I20240502 21:27:03.220938 31393 catalog_manager.cc:5576] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933) does not have the latest schema for tablet 654b4e1fe654455b855b7dae1203badf (table rebalancer_dynamic_table [id=3880230f55a6456b8789cd24843fd807]). Expected version 1 got 0
I20240502 21:27:03.225235 31392 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=3880230f55a6456b8789cd24843fd807] alter complete (version 1)
I20240502 21:27:03.228215 4801 consensus_queue.cc:1035] T 565f30d5539c4b7ea956dc49d0f9a55e P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:03.281018 5451 tablet.cc:1722] T 654b4e1fe654455b855b7dae1203badf P 2d50f33557944bb480ea32199d9ee4f1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:03.282608 5451 tablet.cc:1620] T 654b4e1fe654455b855b7dae1203badf P 2d50f33557944bb480ea32199d9ee4f1: MemRowSet was empty: no flush needed.
I20240502 21:27:03.284843 31392 catalog_manager.cc:5576] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933) does not have the latest schema for tablet 654b4e1fe654455b855b7dae1203badf (table rebalancer_dynamic_table [id=3880230f55a6456b8789cd24843fd807]). Expected version 1 got 0
I20240502 21:27:03.294665 2092 consensus_queue.cc:237] T 2c737c49a32b4f3ea5ef735fa254d48e P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } }
I20240502 21:27:03.299594 31665 raft_consensus.cc:1261] T 2c737c49a32b4f3ea5ef735fa254d48e P 05f4fac41c4f4959b9b316cb112d0ada [term 2 FOLLOWER]: Refusing update from remote peer 1ade0354c44c4c37ae480c7a94b11e1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 21:27:03.301292 31796 raft_consensus.cc:1261] T 2c737c49a32b4f3ea5ef735fa254d48e P 08d7a0680d244b55b93d17f9ab57b84c [term 2 FOLLOWER]: Refusing update from remote peer 1ade0354c44c4c37ae480c7a94b11e1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 21:27:03.301883 4904 consensus_queue.cc:1035] T 2c737c49a32b4f3ea5ef735fa254d48e P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:03.302752 4904 consensus_queue.cc:1035] T 2c737c49a32b4f3ea5ef735fa254d48e P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:03.304356 4904 raft_consensus.cc:2932] T 2c737c49a32b4f3ea5ef735fa254d48e P 1ade0354c44c4c37ae480c7a94b11e1b [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:27:03.305569 31665 raft_consensus.cc:2932] T 2c737c49a32b4f3ea5ef735fa254d48e P 05f4fac41c4f4959b9b316cb112d0ada [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:27:03.306217 31796 raft_consensus.cc:2932] T 2c737c49a32b4f3ea5ef735fa254d48e P 08d7a0680d244b55b93d17f9ab57b84c [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:27:03.307109 5452 ts_tablet_manager.cc:902] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 9dec645b473f40578e726f3e50e219bf: Initiating tablet copy from peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:27:03.308727 31381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2c737c49a32b4f3ea5ef735fa254d48e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:03.312156 31389 catalog_manager.cc:5484] T 2c737c49a32b4f3ea5ef735fa254d48e P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: config changed from index 3 to 5, VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) evicted. New cstate: current_term: 2 leader_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:03.318921 1814 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c737c49a32b4f3ea5ef735fa254d48e with delete_type TABLET_DATA_TOMBSTONED (TS 1e04157b98dd4fad98858c290b73c3db not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47510
I20240502 21:27:03.325219 31389 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_extra_table_40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:03.332630 5457 tablet_replica.cc:330] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db: stopping tablet replica
I20240502 21:27:03.334471 5457 raft_consensus.cc:2229] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:03.334991 5457 raft_consensus.cc:2258] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:03.336004 1946 tablet_service.cc:1467] Processing CreateTablet for tablet 58be3e32928043e1af387efd99e43b63 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=a0f749b890cc4d2da238ffa9f9d97363]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:03.336426 1946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58be3e32928043e1af387efd99e43b63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.337920 31647 tablet_service.cc:1467] Processing CreateTablet for tablet 58be3e32928043e1af387efd99e43b63 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=a0f749b890cc4d2da238ffa9f9d97363]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:03.338265 31647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58be3e32928043e1af387efd99e43b63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.338166 31906 tablet_service.cc:1467] Processing CreateTablet for tablet f1815f008fc041adb2da8c1e626a354f (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=a0f749b890cc4d2da238ffa9f9d97363]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:03.338742 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1815f008fc041adb2da8c1e626a354f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.340749 31645 tablet_service.cc:1467] Processing CreateTablet for tablet f1815f008fc041adb2da8c1e626a354f (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=a0f749b890cc4d2da238ffa9f9d97363]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:03.341064 31645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1815f008fc041adb2da8c1e626a354f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.341842 1945 tablet_service.cc:1467] Processing CreateTablet for tablet 41029f1332c64ba58ee0ee45714ba2a8 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=a0f749b890cc4d2da238ffa9f9d97363]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:03.342154 1945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41029f1332c64ba58ee0ee45714ba2a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.345101 31516 tablet_service.cc:1467] Processing CreateTablet for tablet 58be3e32928043e1af387efd99e43b63 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=a0f749b890cc4d2da238ffa9f9d97363]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:03.345508 31516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58be3e32928043e1af387efd99e43b63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.350018 31905 tablet_service.cc:1467] Processing CreateTablet for tablet 41029f1332c64ba58ee0ee45714ba2a8 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=a0f749b890cc4d2da238ffa9f9d97363]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:03.350353 31905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41029f1332c64ba58ee0ee45714ba2a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.354650 31386 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:03.362632 2077 tablet_service.cc:1467] Processing CreateTablet for tablet 41029f1332c64ba58ee0ee45714ba2a8 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=a0f749b890cc4d2da238ffa9f9d97363]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:03.362994 5460 tablet_bootstrap.cc:492] T f1815f008fc041adb2da8c1e626a354f P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:27:03.363122 2077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41029f1332c64ba58ee0ee45714ba2a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.363940 1814 tablet_service.cc:1467] Processing CreateTablet for tablet f1815f008fc041adb2da8c1e626a354f (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=a0f749b890cc4d2da238ffa9f9d97363]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:03.366454 1814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1815f008fc041adb2da8c1e626a354f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.367259 5460 tablet_bootstrap.cc:654] T f1815f008fc041adb2da8c1e626a354f P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:03.372681 5437 tablet.cc:1722] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:03.373082 5437 tablet.cc:1620] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada: MemRowSet was empty: no flush needed.
I20240502 21:27:03.373423 5438 tablet.cc:1722] T 654b4e1fe654455b855b7dae1203badf P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:03.373838 5438 tablet.cc:1620] T 654b4e1fe654455b855b7dae1203badf P 9dec645b473f40578e726f3e50e219bf: MemRowSet was empty: no flush needed.
I20240502 21:27:03.384366 5461 ts_tablet_manager.cc:902] T 4bc968ae4dcc4636a49257b6d5824bee P 05f4fac41c4f4959b9b316cb112d0ada: Initiating tablet copy from peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:27:03.385753 31393 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=3880230f55a6456b8789cd24843fd807] alter complete (version 2)
I20240502 21:27:03.386705 5462 tablet_bootstrap.cc:492] T 41029f1332c64ba58ee0ee45714ba2a8 P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:27:03.396682 5451 tablet.cc:1722] T 654b4e1fe654455b855b7dae1203badf P 2d50f33557944bb480ea32199d9ee4f1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:03.397182 5451 tablet.cc:1620] T 654b4e1fe654455b855b7dae1203badf P 2d50f33557944bb480ea32199d9ee4f1: MemRowSet was empty: no flush needed.
I20240502 21:27:03.398171 5462 tablet_bootstrap.cc:654] T 41029f1332c64ba58ee0ee45714ba2a8 P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:03.399674 5457 ts_tablet_manager.cc:1875] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:03.406810 5470 tablet_bootstrap.cc:492] T f1815f008fc041adb2da8c1e626a354f P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:27:03.412160 5465 tablet_bootstrap.cc:492] T f1815f008fc041adb2da8c1e626a354f P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:27:03.413619 5465 tablet_bootstrap.cc:654] T f1815f008fc041adb2da8c1e626a354f P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:03.414778 5467 tablet_bootstrap.cc:492] T 58be3e32928043e1af387efd99e43b63 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:27:03.416777 5464 ts_tablet_manager.cc:902] T 883c729aa26f47178f40a17ad3b7e590 P 2d50f33557944bb480ea32199d9ee4f1: Initiating tablet copy from peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:27:03.418200 5476 ts_tablet_manager.cc:902] T b8db5de7df864ab983504e96fb738106 P 05f4fac41c4f4959b9b316cb112d0ada: Initiating tablet copy from peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:27:03.420933 5469 tablet_bootstrap.cc:492] T 41029f1332c64ba58ee0ee45714ba2a8 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:27:03.422686 5469 tablet_bootstrap.cc:654] T 41029f1332c64ba58ee0ee45714ba2a8 P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:03.424739 5470 tablet_bootstrap.cc:654] T f1815f008fc041adb2da8c1e626a354f P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:03.427114 5464 tablet_copy_client.cc:308] T 883c729aa26f47178f40a17ad3b7e590 P 2d50f33557944bb480ea32199d9ee4f1: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.136:34907
I20240502 21:27:03.427749 2251 tablet_copy_service.cc:140] P d1406e92aec84ac585304e57bd37a40d: Received BeginTabletCopySession request for tablet 883c729aa26f47178f40a17ad3b7e590 from peer 2d50f33557944bb480ea32199d9ee4f1 ({username='slave'} at 127.24.226.132:59119)
I20240502 21:27:03.427834 5460 tablet_bootstrap.cc:492] T f1815f008fc041adb2da8c1e626a354f P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:27:03.427915 2251 tablet_copy_service.cc:161] P d1406e92aec84ac585304e57bd37a40d: Beginning new tablet copy session on tablet 883c729aa26f47178f40a17ad3b7e590 from peer 2d50f33557944bb480ea32199d9ee4f1 at {username='slave'} at 127.24.226.132:59119: session id = 2d50f33557944bb480ea32199d9ee4f1-883c729aa26f47178f40a17ad3b7e590
I20240502 21:27:03.427958 5460 ts_tablet_manager.cc:1367] T f1815f008fc041adb2da8c1e626a354f P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.065s user 0.016s sys 0.029s
I20240502 21:27:03.428521 5460 raft_consensus.cc:348] T f1815f008fc041adb2da8c1e626a354f P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:03.428675 5460 raft_consensus.cc:374] T f1815f008fc041adb2da8c1e626a354f P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:03.428728 5460 raft_consensus.cc:729] T f1815f008fc041adb2da8c1e626a354f P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:27:03.428958 5460 consensus_queue.cc:260] T f1815f008fc041adb2da8c1e626a354f P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:03.429486 5460 ts_tablet_manager.cc:1398] T f1815f008fc041adb2da8c1e626a354f P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 21:27:03.429975 5460 tablet_bootstrap.cc:492] T 41029f1332c64ba58ee0ee45714ba2a8 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:27:03.430874 2251 tablet_copy_source_session.cc:205] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:03.431264 5460 tablet_bootstrap.cc:654] T 41029f1332c64ba58ee0ee45714ba2a8 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:03.432498 1965 raft_consensus.cc:1261] T 3e6657723fe84e749dea768de44638e7 P 9dec645b473f40578e726f3e50e219bf [term 2 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:27:03.433101 5089 consensus_queue.cc:1035] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:03.433966 5464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 883c729aa26f47178f40a17ad3b7e590. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.438459 5467 tablet_bootstrap.cc:654] T 58be3e32928043e1af387efd99e43b63 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:03.441371 5465 tablet_bootstrap.cc:492] T f1815f008fc041adb2da8c1e626a354f P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:27:03.441509 5460 tablet_bootstrap.cc:492] T 41029f1332c64ba58ee0ee45714ba2a8 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:27:03.441589 5465 ts_tablet_manager.cc:1367] T f1815f008fc041adb2da8c1e626a354f P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.003s
I20240502 21:27:03.442914 5460 ts_tablet_manager.cc:1367] T 41029f1332c64ba58ee0ee45714ba2a8 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.003s
I20240502 21:27:03.443710 5460 raft_consensus.cc:348] T 41029f1332c64ba58ee0ee45714ba2a8 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:03.444963 5460 raft_consensus.cc:374] T 41029f1332c64ba58ee0ee45714ba2a8 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:03.445118 5460 raft_consensus.cc:729] T 41029f1332c64ba58ee0ee45714ba2a8 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:27:03.445430 1833 raft_consensus.cc:1261] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [term 2 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:27:03.445457 5460 consensus_queue.cc:260] T 41029f1332c64ba58ee0ee45714ba2a8 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:03.446214 4942 consensus_queue.cc:1035] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:03.447675 5465 raft_consensus.cc:348] T f1815f008fc041adb2da8c1e626a354f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:03.448232 5467 tablet_bootstrap.cc:492] T 58be3e32928043e1af387efd99e43b63 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:27:03.446321 5460 ts_tablet_manager.cc:1398] T 41029f1332c64ba58ee0ee45714ba2a8 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:27:03.454991 5467 ts_tablet_manager.cc:1367] T 58be3e32928043e1af387efd99e43b63 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.000s
I20240502 21:27:03.455682 5467 raft_consensus.cc:348] T 58be3e32928043e1af387efd99e43b63 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:03.455885 5467 raft_consensus.cc:374] T 58be3e32928043e1af387efd99e43b63 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:03.455938 5467 raft_consensus.cc:729] T 58be3e32928043e1af387efd99e43b63 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:27:03.456105 5467 consensus_queue.cc:260] T 58be3e32928043e1af387efd99e43b63 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:03.456590 5467 ts_tablet_manager.cc:1398] T 58be3e32928043e1af387efd99e43b63 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:03.460345 5469 tablet_bootstrap.cc:492] T 41029f1332c64ba58ee0ee45714ba2a8 P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:27:03.458968 5465 raft_consensus.cc:374] T f1815f008fc041adb2da8c1e626a354f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:03.460557 5476 tablet_copy_client.cc:308] T b8db5de7df864ab983504e96fb738106 P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.131:40807
I20240502 21:27:03.460623 5465 raft_consensus.cc:729] T f1815f008fc041adb2da8c1e626a354f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:27:03.461251 5469 ts_tablet_manager.cc:1367] T 41029f1332c64ba58ee0ee45714ba2a8 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.000s
I20240502 21:27:03.461326 31817 tablet_copy_service.cc:140] P 08d7a0680d244b55b93d17f9ab57b84c: Received BeginTabletCopySession request for tablet b8db5de7df864ab983504e96fb738106 from peer 05f4fac41c4f4959b9b316cb112d0ada ({username='slave'} at 127.24.226.130:55955)
I20240502 21:27:03.461504 31817 tablet_copy_service.cc:161] P 08d7a0680d244b55b93d17f9ab57b84c: Beginning new tablet copy session on tablet b8db5de7df864ab983504e96fb738106 from peer 05f4fac41c4f4959b9b316cb112d0ada at {username='slave'} at 127.24.226.130:55955: session id = 05f4fac41c4f4959b9b316cb112d0ada-b8db5de7df864ab983504e96fb738106
I20240502 21:27:03.462095 5465 consensus_queue.cc:260] T f1815f008fc041adb2da8c1e626a354f P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:03.463855 5469 raft_consensus.cc:348] T 41029f1332c64ba58ee0ee45714ba2a8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:03.464036 5469 raft_consensus.cc:374] T 41029f1332c64ba58ee0ee45714ba2a8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:03.464121 5469 raft_consensus.cc:729] T 41029f1332c64ba58ee0ee45714ba2a8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:27:03.464371 5469 consensus_queue.cc:260] T 41029f1332c64ba58ee0ee45714ba2a8 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:03.465106 31817 tablet_copy_source_session.cc:205] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:03.466439 5476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8db5de7df864ab983504e96fb738106. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.469732 5465 ts_tablet_manager.cc:1398] T f1815f008fc041adb2da8c1e626a354f P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.028s user 0.000s sys 0.001s
I20240502 21:27:03.470024 5464 tablet_copy_client.cc:783] T 883c729aa26f47178f40a17ad3b7e590 P 2d50f33557944bb480ea32199d9ee4f1: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:03.470292 5464 tablet_copy_client.cc:647] T 883c729aa26f47178f40a17ad3b7e590 P 2d50f33557944bb480ea32199d9ee4f1: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:03.472986 5457 ts_tablet_manager.cc:1888] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 21:27:03.475369 5465 tablet_bootstrap.cc:492] T 58be3e32928043e1af387efd99e43b63 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:27:03.476385 5464 tablet_copy_client.cc:523] T 883c729aa26f47178f40a17ad3b7e590 P 2d50f33557944bb480ea32199d9ee4f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:03.478394 5465 tablet_bootstrap.cc:654] T 58be3e32928043e1af387efd99e43b63 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:03.478638 5457 log.cc:1196] T 2c737c49a32b4f3ea5ef735fa254d48e P 1e04157b98dd4fad98858c290b73c3db: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal/wals/2c737c49a32b4f3ea5ef735fa254d48e
I20240502 21:27:03.479682 31381 catalog_manager.cc:4830] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): tablet 2c737c49a32b4f3ea5ef735fa254d48e (table rebalancer_extra_table_18 [id=a35acd1fa3294955995e9c2c846a5de0]) successfully deleted
I20240502 21:27:03.481907 5464 tablet_bootstrap.cc:492] T 883c729aa26f47178f40a17ad3b7e590 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:27:03.480284 5469 ts_tablet_manager.cc:1398] T 41029f1332c64ba58ee0ee45714ba2a8 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240502 21:27:03.475955 5476 tablet_copy_client.cc:783] T b8db5de7df864ab983504e96fb738106 P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:03.482936 5476 tablet_copy_client.cc:647] T b8db5de7df864ab983504e96fb738106 P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:03.484800 5476 tablet_copy_client.cc:523] T b8db5de7df864ab983504e96fb738106 P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:03.486043 5462 tablet_bootstrap.cc:492] T 41029f1332c64ba58ee0ee45714ba2a8 P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:27:03.486227 5462 ts_tablet_manager.cc:1367] T 41029f1332c64ba58ee0ee45714ba2a8 P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.100s user 0.003s sys 0.003s
I20240502 21:27:03.486974 5462 raft_consensus.cc:348] T 41029f1332c64ba58ee0ee45714ba2a8 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:03.487248 5462 raft_consensus.cc:374] T 41029f1332c64ba58ee0ee45714ba2a8 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:03.487365 5462 raft_consensus.cc:729] T 41029f1332c64ba58ee0ee45714ba2a8 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:27:03.487670 5462 consensus_queue.cc:260] T 41029f1332c64ba58ee0ee45714ba2a8 P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:03.488157 5476 tablet_bootstrap.cc:492] T b8db5de7df864ab983504e96fb738106 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:27:03.490218 5465 tablet_bootstrap.cc:492] T 58be3e32928043e1af387efd99e43b63 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:27:03.490336 5465 ts_tablet_manager.cc:1367] T 58be3e32928043e1af387efd99e43b63 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.003s
I20240502 21:27:03.490986 5465 raft_consensus.cc:348] T 58be3e32928043e1af387efd99e43b63 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:03.491171 5465 raft_consensus.cc:374] T 58be3e32928043e1af387efd99e43b63 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:03.491223 5465 raft_consensus.cc:729] T 58be3e32928043e1af387efd99e43b63 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:27:03.491442 5465 consensus_queue.cc:260] T 58be3e32928043e1af387efd99e43b63 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:03.491469 31389 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:03.496224 5492 ts_tablet_manager.cc:902] T 1f743c7345184feaa50871dc49a1f136 P 08d7a0680d244b55b93d17f9ab57b84c: Initiating tablet copy from peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:27:03.497678 5465 ts_tablet_manager.cc:1398] T 58be3e32928043e1af387efd99e43b63 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 21:27:03.520736 2094 raft_consensus.cc:1261] T 3e6657723fe84e749dea768de44638e7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 2 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:27:03.522403 4942 consensus_queue.cc:1035] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:03.523396 5464 tablet_bootstrap.cc:492] T 883c729aa26f47178f40a17ad3b7e590 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:03.523814 5464 tablet_bootstrap.cc:492] T 883c729aa26f47178f40a17ad3b7e590 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap complete.
I20240502 21:27:03.524056 5464 ts_tablet_manager.cc:1367] T 883c729aa26f47178f40a17ad3b7e590 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.042s user 0.008s sys 0.000s
I20240502 21:27:03.524832 5464 raft_consensus.cc:348] T 883c729aa26f47178f40a17ad3b7e590 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } }
I20240502 21:27:03.525111 5464 raft_consensus.cc:729] T 883c729aa26f47178f40a17ad3b7e590 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: LEARNER
I20240502 21:27:03.529996 5437 tablet.cc:1722] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:03.525377 5464 consensus_queue.cc:260] T 883c729aa26f47178f40a17ad3b7e590 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: NON_VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: true } }
I20240502 21:27:03.534037 5451 tablet.cc:1722] T 654b4e1fe654455b855b7dae1203badf P 2d50f33557944bb480ea32199d9ee4f1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:03.537402 5473 ts_tablet_manager.cc:902] T f61e34e65429408e83a13c7bc743e78a P 1ade0354c44c4c37ae480c7a94b11e1b: Initiating tablet copy from peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:27:03.540496 5464 ts_tablet_manager.cc:1398] T 883c729aa26f47178f40a17ad3b7e590 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.014s user 0.004s sys 0.000s
I20240502 21:27:03.544410 5470 tablet_bootstrap.cc:492] T f1815f008fc041adb2da8c1e626a354f P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:27:03.545689 2251 tablet_copy_service.cc:342] P d1406e92aec84ac585304e57bd37a40d: Request end of tablet copy session 2d50f33557944bb480ea32199d9ee4f1-883c729aa26f47178f40a17ad3b7e590 received from {username='slave'} at 127.24.226.132:59119
I20240502 21:27:03.545864 2251 tablet_copy_service.cc:433] P d1406e92aec84ac585304e57bd37a40d: ending tablet copy session 2d50f33557944bb480ea32199d9ee4f1-883c729aa26f47178f40a17ad3b7e590 on tablet 883c729aa26f47178f40a17ad3b7e590 with peer 2d50f33557944bb480ea32199d9ee4f1
I20240502 21:27:03.548679 5470 ts_tablet_manager.cc:1367] T f1815f008fc041adb2da8c1e626a354f P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.142s user 0.003s sys 0.001s
I20240502 21:27:03.549392 31392 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table_new_name [id=3880230f55a6456b8789cd24843fd807] alter complete (version 3)
I20240502 21:27:03.550814 5470 raft_consensus.cc:348] T f1815f008fc041adb2da8c1e626a354f P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:03.496296 5452 tablet_copy_client.cc:308] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 9dec645b473f40578e726f3e50e219bf: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.129:41039
I20240502 21:27:03.552681 31556 tablet_copy_service.cc:140] P a44f41b0b6644fb6aa29476aed8c964f: Received BeginTabletCopySession request for tablet 5bc6c7211d5f4f73b2a050b9ec78c6b1 from peer 9dec645b473f40578e726f3e50e219bf ({username='slave'} at 127.24.226.134:40609)
I20240502 21:27:03.552820 31556 tablet_copy_service.cc:161] P a44f41b0b6644fb6aa29476aed8c964f: Beginning new tablet copy session on tablet 5bc6c7211d5f4f73b2a050b9ec78c6b1 from peer 9dec645b473f40578e726f3e50e219bf at {username='slave'} at 127.24.226.134:40609: session id = 9dec645b473f40578e726f3e50e219bf-5bc6c7211d5f4f73b2a050b9ec78c6b1
I20240502 21:27:03.554003 5462 ts_tablet_manager.cc:1398] T 41029f1332c64ba58ee0ee45714ba2a8 P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.068s user 0.001s sys 0.001s
I20240502 21:27:03.554252 5462 tablet_bootstrap.cc:492] T 58be3e32928043e1af387efd99e43b63 P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:27:03.554612 5461 tablet_copy_client.cc:308] T 4bc968ae4dcc4636a49257b6d5824bee P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.129:41039
I20240502 21:27:03.555371 31556 tablet_copy_source_session.cc:205] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:03.556874 31556 tablet_copy_service.cc:140] P a44f41b0b6644fb6aa29476aed8c964f: Received BeginTabletCopySession request for tablet 4bc968ae4dcc4636a49257b6d5824bee from peer 05f4fac41c4f4959b9b316cb112d0ada ({username='slave'} at 127.24.226.130:41329)
I20240502 21:27:03.557000 31556 tablet_copy_service.cc:161] P a44f41b0b6644fb6aa29476aed8c964f: Beginning new tablet copy session on tablet 4bc968ae4dcc4636a49257b6d5824bee from peer 05f4fac41c4f4959b9b316cb112d0ada at {username='slave'} at 127.24.226.130:41329: session id = 05f4fac41c4f4959b9b316cb112d0ada-4bc968ae4dcc4636a49257b6d5824bee
I20240502 21:27:03.557600 5452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc6c7211d5f4f73b2a050b9ec78c6b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.560024 31556 tablet_copy_source_session.cc:205] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:03.568092 5452 tablet_copy_client.cc:783] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 9dec645b473f40578e726f3e50e219bf: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:03.568300 5452 tablet_copy_client.cc:647] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 9dec645b473f40578e726f3e50e219bf: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:03.570590 5461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bc968ae4dcc4636a49257b6d5824bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.562752 5510 ts_tablet_manager.cc:902] T 381ac1d714204cef86c8e1a73e24b863 P 1e04157b98dd4fad98858c290b73c3db: Initiating tablet copy from peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:27:03.576952 5452 tablet_copy_client.cc:523] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 9dec645b473f40578e726f3e50e219bf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:03.580000 5452 tablet_bootstrap.cc:492] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:27:03.589725 5470 raft_consensus.cc:374] T f1815f008fc041adb2da8c1e626a354f P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:03.590121 5470 raft_consensus.cc:729] T f1815f008fc041adb2da8c1e626a354f P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:27:03.590710 5470 consensus_queue.cc:260] T f1815f008fc041adb2da8c1e626a354f P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:03.600647 5470 ts_tablet_manager.cc:1398] T f1815f008fc041adb2da8c1e626a354f P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.051s user 0.001s sys 0.001s
I20240502 21:27:03.596369 5510 tablet_copy_client.cc:308] T 381ac1d714204cef86c8e1a73e24b863 P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.129:41039
I20240502 21:27:03.607124 31556 tablet_copy_service.cc:140] P a44f41b0b6644fb6aa29476aed8c964f: Received BeginTabletCopySession request for tablet 381ac1d714204cef86c8e1a73e24b863 from peer 1e04157b98dd4fad98858c290b73c3db ({username='slave'} at 127.24.226.133:49939)
I20240502 21:27:03.607323 31556 tablet_copy_service.cc:161] P a44f41b0b6644fb6aa29476aed8c964f: Beginning new tablet copy session on tablet 381ac1d714204cef86c8e1a73e24b863 from peer 1e04157b98dd4fad98858c290b73c3db at {username='slave'} at 127.24.226.133:49939: session id = 1e04157b98dd4fad98858c290b73c3db-381ac1d714204cef86c8e1a73e24b863
I20240502 21:27:03.608222 5462 tablet_bootstrap.cc:654] T 58be3e32928043e1af387efd99e43b63 P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:03.614490 31556 tablet_copy_source_session.cc:205] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:03.622428 5510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 381ac1d714204cef86c8e1a73e24b863. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.629460 5438 tablet.cc:1722] T 654b4e1fe654455b855b7dae1203badf P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:03.634272 5510 tablet_copy_client.cc:783] T 381ac1d714204cef86c8e1a73e24b863 P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:03.634507 5510 tablet_copy_client.cc:647] T 381ac1d714204cef86c8e1a73e24b863 P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:03.636559 5506 raft_consensus.cc:482] T 58be3e32928043e1af387efd99e43b63 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:03.636806 5506 raft_consensus.cc:504] T 58be3e32928043e1af387efd99e43b63 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:03.638082 5506 leader_election.cc:288] T 58be3e32928043e1af387efd99e43b63 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:27:03.640256 1965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58be3e32928043e1af387efd99e43b63" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
W20240502 21:27:03.640913 31469 leader_election.cc:341] T 58be3e32928043e1af387efd99e43b63 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:03.641602 5476 tablet_bootstrap.cc:492] T b8db5de7df864ab983504e96fb738106 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:03.641953 5476 tablet_bootstrap.cc:492] T b8db5de7df864ab983504e96fb738106 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap complete.
I20240502 21:27:03.642177 5476 ts_tablet_manager.cc:1367] T b8db5de7df864ab983504e96fb738106 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.154s user 0.022s sys 0.014s
I20240502 21:27:03.642704 5476 raft_consensus.cc:348] T b8db5de7df864ab983504e96fb738106 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } }
I20240502 21:27:03.642896 5476 raft_consensus.cc:729] T b8db5de7df864ab983504e96fb738106 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: LEARNER
I20240502 21:27:03.643059 5476 consensus_queue.cc:260] T b8db5de7df864ab983504e96fb738106 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } }
I20240502 21:27:03.645205 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58be3e32928043e1af387efd99e43b63" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:27:03.645452 31665 raft_consensus.cc:2454] T 58be3e32928043e1af387efd99e43b63 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:27:03.648907 5476 ts_tablet_manager.cc:1398] T b8db5de7df864ab983504e96fb738106 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 21:27:03.649827 31817 tablet_copy_service.cc:342] P 08d7a0680d244b55b93d17f9ab57b84c: Request end of tablet copy session 05f4fac41c4f4959b9b316cb112d0ada-b8db5de7df864ab983504e96fb738106 received from {username='slave'} at 127.24.226.130:55955
I20240502 21:27:03.649878 5512 raft_consensus.cc:482] T f1815f008fc041adb2da8c1e626a354f P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:03.649958 31817 tablet_copy_service.cc:433] P 08d7a0680d244b55b93d17f9ab57b84c: ending tablet copy session 05f4fac41c4f4959b9b316cb112d0ada-b8db5de7df864ab983504e96fb738106 on tablet b8db5de7df864ab983504e96fb738106 with peer 05f4fac41c4f4959b9b316cb112d0ada
I20240502 21:27:03.650010 5512 raft_consensus.cc:504] T f1815f008fc041adb2da8c1e626a354f P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:03.651268 5512 leader_election.cc:288] T f1815f008fc041adb2da8c1e626a354f P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:27:03.651381 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1815f008fc041adb2da8c1e626a354f" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:27:03.651546 31665 raft_consensus.cc:2454] T f1815f008fc041adb2da8c1e626a354f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:27:03.651612 1832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1815f008fc041adb2da8c1e626a354f" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:27:03.651808 1832 raft_consensus.cc:2454] T f1815f008fc041adb2da8c1e626a354f P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:27:03.652180 31862 leader_election.cc:302] T f1815f008fc041adb2da8c1e626a354f P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:27:03.652483 5512 raft_consensus.cc:2781] T f1815f008fc041adb2da8c1e626a354f P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:03.652608 5512 raft_consensus.cc:482] T f1815f008fc041adb2da8c1e626a354f P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:03.652688 5512 raft_consensus.cc:3037] T f1815f008fc041adb2da8c1e626a354f P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:03.655632 5512 raft_consensus.cc:504] T f1815f008fc041adb2da8c1e626a354f P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:03.656059 5462 tablet_bootstrap.cc:492] T 58be3e32928043e1af387efd99e43b63 P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:27:03.656620 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1815f008fc041adb2da8c1e626a354f" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:27:03.656813 31665 raft_consensus.cc:3037] T f1815f008fc041adb2da8c1e626a354f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:03.656852 5462 ts_tablet_manager.cc:1367] T 58be3e32928043e1af387efd99e43b63 P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.103s user 0.004s sys 0.000s
I20240502 21:27:03.656903 1832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1815f008fc041adb2da8c1e626a354f" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:27:03.657095 1832 raft_consensus.cc:3037] T f1815f008fc041adb2da8c1e626a354f P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:03.657729 5462 raft_consensus.cc:348] T 58be3e32928043e1af387efd99e43b63 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:03.658061 5462 raft_consensus.cc:374] T 58be3e32928043e1af387efd99e43b63 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:03.658146 5462 raft_consensus.cc:729] T 58be3e32928043e1af387efd99e43b63 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:27:03.658368 5462 consensus_queue.cc:260] T 58be3e32928043e1af387efd99e43b63 P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:03.659500 1832 raft_consensus.cc:2454] T f1815f008fc041adb2da8c1e626a354f P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:27:03.659796 31665 raft_consensus.cc:2454] T f1815f008fc041adb2da8c1e626a354f P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:27:03.659961 31862 leader_election.cc:302] T f1815f008fc041adb2da8c1e626a354f P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:27:03.660238 5202 raft_consensus.cc:2781] T f1815f008fc041adb2da8c1e626a354f P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:03.660645 5462 ts_tablet_manager.cc:1398] T 58be3e32928043e1af387efd99e43b63 P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 21:27:03.669255 5461 tablet_copy_client.cc:783] T 4bc968ae4dcc4636a49257b6d5824bee P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:03.672704 5512 leader_election.cc:288] T f1815f008fc041adb2da8c1e626a354f P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Requested vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:27:03.674484 5202 raft_consensus.cc:686] T f1815f008fc041adb2da8c1e626a354f P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Running, Role: LEADER
I20240502 21:27:03.674850 5202 consensus_queue.cc:237] T f1815f008fc041adb2da8c1e626a354f P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:03.675259 31472 leader_election.cc:302] T 58be3e32928043e1af387efd99e43b63 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, a44f41b0b6644fb6aa29476aed8c964f; no voters: 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:03.675534 4942 raft_consensus.cc:2781] T 58be3e32928043e1af387efd99e43b63 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:03.675633 4942 raft_consensus.cc:482] T 58be3e32928043e1af387efd99e43b63 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:03.675715 4942 raft_consensus.cc:3037] T 58be3e32928043e1af387efd99e43b63 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:03.678575 4942 raft_consensus.cc:504] T 58be3e32928043e1af387efd99e43b63 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:03.681622 1965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58be3e32928043e1af387efd99e43b63" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf"
I20240502 21:27:03.681844 1965 raft_consensus.cc:3037] T 58be3e32928043e1af387efd99e43b63 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:03.684134 4942 leader_election.cc:288] T 58be3e32928043e1af387efd99e43b63 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Requested vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:27:03.684993 1965 raft_consensus.cc:2454] T 58be3e32928043e1af387efd99e43b63 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:27:03.685367 31469 leader_election.cc:302] T 58be3e32928043e1af387efd99e43b63 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dec645b473f40578e726f3e50e219bf, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:27:03.685705 5089 raft_consensus.cc:2781] T 58be3e32928043e1af387efd99e43b63 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:03.685897 5089 raft_consensus.cc:686] T 58be3e32928043e1af387efd99e43b63 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:27:03.686305 5089 consensus_queue.cc:237] T 58be3e32928043e1af387efd99e43b63 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:03.699883 5510 tablet_copy_client.cc:523] T 381ac1d714204cef86c8e1a73e24b863 P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:03.707055 5490 raft_consensus.cc:482] T 41029f1332c64ba58ee0ee45714ba2a8 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:03.707218 5490 raft_consensus.cc:504] T 41029f1332c64ba58ee0ee45714ba2a8 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:03.708014 5490 leader_election.cc:288] T 41029f1332c64ba58ee0ee45714ba2a8 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:27:03.708505 2092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41029f1332c64ba58ee0ee45714ba2a8" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:27:03.708757 2092 raft_consensus.cc:2454] T 41029f1332c64ba58ee0ee45714ba2a8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 0.
I20240502 21:27:03.709395 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41029f1332c64ba58ee0ee45714ba2a8" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:27:03.709630 31925 raft_consensus.cc:2454] T 41029f1332c64ba58ee0ee45714ba2a8 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 0.
I20240502 21:27:03.709712 5473 tablet_copy_client.cc:308] T f61e34e65429408e83a13c7bc743e78a P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.130:37645
I20240502 21:27:03.695997 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58be3e32928043e1af387efd99e43b63" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:27:03.710021 1901 leader_election.cc:302] T 41029f1332c64ba58ee0ee45714ba2a8 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:27:03.710191 31665 raft_consensus.cc:3037] T 58be3e32928043e1af387efd99e43b63 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:03.710901 5492 tablet_copy_client.cc:308] T 1f743c7345184feaa50871dc49a1f136 P 08d7a0680d244b55b93d17f9ab57b84c: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.132:35933
I20240502 21:27:03.711123 31686 tablet_copy_service.cc:140] P 05f4fac41c4f4959b9b316cb112d0ada: Received BeginTabletCopySession request for tablet f61e34e65429408e83a13c7bc743e78a from peer 1ade0354c44c4c37ae480c7a94b11e1b ({username='slave'} at 127.24.226.135:46887)
I20240502 21:27:03.711268 31686 tablet_copy_service.cc:161] P 05f4fac41c4f4959b9b316cb112d0ada: Beginning new tablet copy session on tablet f61e34e65429408e83a13c7bc743e78a from peer 1ade0354c44c4c37ae480c7a94b11e1b at {username='slave'} at 127.24.226.135:46887: session id = 1ade0354c44c4c37ae480c7a94b11e1b-f61e34e65429408e83a13c7bc743e78a
I20240502 21:27:03.712566 31947 tablet_copy_service.cc:140] P 2d50f33557944bb480ea32199d9ee4f1: Received BeginTabletCopySession request for tablet 1f743c7345184feaa50871dc49a1f136 from peer 08d7a0680d244b55b93d17f9ab57b84c ({username='slave'} at 127.24.226.131:54067)
I20240502 21:27:03.712749 31947 tablet_copy_service.cc:161] P 2d50f33557944bb480ea32199d9ee4f1: Beginning new tablet copy session on tablet 1f743c7345184feaa50871dc49a1f136 from peer 08d7a0680d244b55b93d17f9ab57b84c at {username='slave'} at 127.24.226.131:54067: session id = 08d7a0680d244b55b93d17f9ab57b84c-1f743c7345184feaa50871dc49a1f136
I20240502 21:27:03.715068 31665 raft_consensus.cc:2454] T 58be3e32928043e1af387efd99e43b63 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:27:03.715402 5490 raft_consensus.cc:2781] T 41029f1332c64ba58ee0ee45714ba2a8 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:03.715525 5490 raft_consensus.cc:482] T 41029f1332c64ba58ee0ee45714ba2a8 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:03.715641 5490 raft_consensus.cc:3037] T 41029f1332c64ba58ee0ee45714ba2a8 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:03.716199 5461 tablet_copy_client.cc:647] T 4bc968ae4dcc4636a49257b6d5824bee P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:03.716399 31947 tablet_copy_source_session.cc:205] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:03.718920 31686 tablet_copy_source_session.cc:205] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:03.721144 5473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f61e34e65429408e83a13c7bc743e78a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.721562 5492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f743c7345184feaa50871dc49a1f136. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.722810 5461 tablet_copy_client.cc:523] T 4bc968ae4dcc4636a49257b6d5824bee P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:03.729627 5510 tablet_bootstrap.cc:492] T 381ac1d714204cef86c8e1a73e24b863 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:27:03.731923 5461 tablet_bootstrap.cc:492] T 4bc968ae4dcc4636a49257b6d5824bee P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:27:03.733759 31386 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:03.733996 31386 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:03.736362 5452 tablet_bootstrap.cc:492] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 9dec645b473f40578e726f3e50e219bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:03.737995 31386 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 654b4e1fe654455b855b7dae1203badf
I20240502 21:27:03.738955 31645 tablet_service.cc:1514] Processing DeleteTablet for tablet 654b4e1fe654455b855b7dae1203badf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:03 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:27:03.739415 31906 tablet_service.cc:1514] Processing DeleteTablet for tablet 654b4e1fe654455b855b7dae1203badf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:03 UTC) from {username='slave'} at 127.0.0.1:47824
I20240502 21:27:03.740417 1945 tablet_service.cc:1514] Processing DeleteTablet for tablet 654b4e1fe654455b855b7dae1203badf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:03 UTC) from {username='slave'} at 127.0.0.1:33002
I20240502 21:27:03.747426 5490 raft_consensus.cc:504] T 41029f1332c64ba58ee0ee45714ba2a8 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:03.748417 5490 leader_election.cc:288] T 41029f1332c64ba58ee0ee45714ba2a8 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 election: Requested vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:27:03.748821 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41029f1332c64ba58ee0ee45714ba2a8" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:27:03.749065 31925 raft_consensus.cc:3037] T 41029f1332c64ba58ee0ee45714ba2a8 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:03.749115 2092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41029f1332c64ba58ee0ee45714ba2a8" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:27:03.749321 2092 raft_consensus.cc:3037] T 41029f1332c64ba58ee0ee45714ba2a8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:03.751029 5452 tablet_bootstrap.cc:492] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 9dec645b473f40578e726f3e50e219bf: Bootstrap complete.
I20240502 21:27:03.751278 5452 ts_tablet_manager.cc:1367] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.171s user 0.008s sys 0.006s
I20240502 21:27:03.751828 2092 raft_consensus.cc:2454] T 41029f1332c64ba58ee0ee45714ba2a8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 1.
I20240502 21:27:03.751993 31925 raft_consensus.cc:2454] T 41029f1332c64ba58ee0ee45714ba2a8 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 1.
I20240502 21:27:03.751996 5452 raft_consensus.cc:348] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:27:03.752231 5452 raft_consensus.cc:729] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: LEARNER
I20240502 21:27:03.752424 1902 leader_election.cc:302] T 41029f1332c64ba58ee0ee45714ba2a8 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:27:03.758858 31386 catalog_manager.cc:5484] T f1815f008fc041adb2da8c1e626a354f P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132). New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:03.760994 5490 raft_consensus.cc:2781] T 41029f1332c64ba58ee0ee45714ba2a8 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:03.761699 5490 raft_consensus.cc:686] T 41029f1332c64ba58ee0ee45714ba2a8 P 9dec645b473f40578e726f3e50e219bf [term 1 LEADER]: Becoming Leader. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Running, Role: LEADER
I20240502 21:27:03.762005 5490 consensus_queue.cc:237] T 41029f1332c64ba58ee0ee45714ba2a8 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:03.763221 5452 consensus_queue.cc:260] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:27:03.764305 31388 catalog_manager.cc:5484] T 58be3e32928043e1af387efd99e43b63 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 0 to 1, leader changed from <none> to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:03.769133 5452 ts_tablet_manager.cc:1398] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240502 21:27:03.770076 5492 tablet_copy_client.cc:783] T 1f743c7345184feaa50871dc49a1f136 P 08d7a0680d244b55b93d17f9ab57b84c: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:03.770828 31556 tablet_copy_service.cc:342] P a44f41b0b6644fb6aa29476aed8c964f: Request end of tablet copy session 9dec645b473f40578e726f3e50e219bf-5bc6c7211d5f4f73b2a050b9ec78c6b1 received from {username='slave'} at 127.24.226.134:40609
I20240502 21:27:03.770956 31556 tablet_copy_service.cc:433] P a44f41b0b6644fb6aa29476aed8c964f: ending tablet copy session 9dec645b473f40578e726f3e50e219bf-5bc6c7211d5f4f73b2a050b9ec78c6b1 on tablet 5bc6c7211d5f4f73b2a050b9ec78c6b1 with peer 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:03.772989 5492 tablet_copy_client.cc:647] T 1f743c7345184feaa50871dc49a1f136 P 08d7a0680d244b55b93d17f9ab57b84c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:03.774490 5492 tablet_copy_client.cc:523] T 1f743c7345184feaa50871dc49a1f136 P 08d7a0680d244b55b93d17f9ab57b84c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:03.780611 5492 tablet_bootstrap.cc:492] T 1f743c7345184feaa50871dc49a1f136 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:27:03.782459 5535 tablet_replica.cc:330] T 654b4e1fe654455b855b7dae1203badf P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:27:03.786145 5534 tablet_replica.cc:330] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:27:03.786451 5535 raft_consensus.cc:2229] T 654b4e1fe654455b855b7dae1203badf P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:03.786772 5534 raft_consensus.cc:2229] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:03.787142 5535 raft_consensus.cc:2258] T 654b4e1fe654455b855b7dae1203badf P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:03.788784 5534 raft_consensus.cc:2258] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:03.791057 5535 ts_tablet_manager.cc:1875] T 654b4e1fe654455b855b7dae1203badf P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:03.795619 5492 tablet_bootstrap.cc:492] T 1f743c7345184feaa50871dc49a1f136 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:03.795874 5534 ts_tablet_manager.cc:1875] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:03.796950 5492 tablet_bootstrap.cc:492] T 1f743c7345184feaa50871dc49a1f136 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap complete.
I20240502 21:27:03.797539 5492 ts_tablet_manager.cc:1367] T 1f743c7345184feaa50871dc49a1f136 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.002s
I20240502 21:27:03.798441 5492 raft_consensus.cc:348] T 1f743c7345184feaa50871dc49a1f136 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } }
I20240502 21:27:03.799499 5492 raft_consensus.cc:729] T 1f743c7345184feaa50871dc49a1f136 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: LEARNER
I20240502 21:27:03.799883 5492 consensus_queue.cc:260] T 1f743c7345184feaa50871dc49a1f136 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: NON_VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: true } }
I20240502 21:27:03.800956 5492 ts_tablet_manager.cc:1398] T 1f743c7345184feaa50871dc49a1f136 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:27:03.801971 31947 tablet_copy_service.cc:342] P 2d50f33557944bb480ea32199d9ee4f1: Request end of tablet copy session 08d7a0680d244b55b93d17f9ab57b84c-1f743c7345184feaa50871dc49a1f136 received from {username='slave'} at 127.24.226.131:54067
I20240502 21:27:03.802106 31947 tablet_copy_service.cc:433] P 2d50f33557944bb480ea32199d9ee4f1: ending tablet copy session 08d7a0680d244b55b93d17f9ab57b84c-1f743c7345184feaa50871dc49a1f136 on tablet 1f743c7345184feaa50871dc49a1f136 with peer 08d7a0680d244b55b93d17f9ab57b84c
I20240502 21:27:03.820093 5473 tablet_copy_client.cc:783] T f61e34e65429408e83a13c7bc743e78a P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:03.821185 5535 ts_tablet_manager.cc:1888] T 654b4e1fe654455b855b7dae1203badf P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:03.825278 5534 ts_tablet_manager.cc:1888] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:03.825474 5535 log.cc:1196] T 654b4e1fe654455b855b7dae1203badf P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/654b4e1fe654455b855b7dae1203badf
I20240502 21:27:03.826056 5535 ts_tablet_manager.cc:1909] T 654b4e1fe654455b855b7dae1203badf P 2d50f33557944bb480ea32199d9ee4f1: Deleting consensus metadata
I20240502 21:27:03.826222 5534 log.cc:1196] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/654b4e1fe654455b855b7dae1203badf
I20240502 21:27:03.826598 5534 ts_tablet_manager.cc:1909] T 654b4e1fe654455b855b7dae1203badf P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:27:03.829011 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet 654b4e1fe654455b855b7dae1203badf (table rebalancer_dynamic_table_new_name [id=3880230f55a6456b8789cd24843fd807]) successfully deleted
I20240502 21:27:03.829165 5473 tablet_copy_client.cc:647] T f61e34e65429408e83a13c7bc743e78a P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:03.830080 5536 tablet_replica.cc:330] T 654b4e1fe654455b855b7dae1203badf P 9dec645b473f40578e726f3e50e219bf: stopping tablet replica
I20240502 21:27:03.830006 31388 catalog_manager.cc:5484] T 41029f1332c64ba58ee0ee45714ba2a8 P 9dec645b473f40578e726f3e50e219bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dec645b473f40578e726f3e50e219bf (127.24.226.134). New cstate: current_term: 1 leader_uuid: "9dec645b473f40578e726f3e50e219bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:03.830631 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet 654b4e1fe654455b855b7dae1203badf (table rebalancer_dynamic_table_new_name [id=3880230f55a6456b8789cd24843fd807]) successfully deleted
I20240502 21:27:03.830672 5536 raft_consensus.cc:2229] T 654b4e1fe654455b855b7dae1203badf P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:03.831895 5536 raft_consensus.cc:2258] T 654b4e1fe654455b855b7dae1203badf P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:03.833151 5536 ts_tablet_manager.cc:1875] T 654b4e1fe654455b855b7dae1203badf P 9dec645b473f40578e726f3e50e219bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:03.834898 5473 tablet_copy_client.cc:523] T f61e34e65429408e83a13c7bc743e78a P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:03.841176 31388 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:03.841470 5461 tablet_bootstrap.cc:492] T 4bc968ae4dcc4636a49257b6d5824bee P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:03.841814 5461 tablet_bootstrap.cc:492] T 4bc968ae4dcc4636a49257b6d5824bee P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap complete.
I20240502 21:27:03.842031 5461 ts_tablet_manager.cc:1367] T 4bc968ae4dcc4636a49257b6d5824bee P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.110s user 0.004s sys 0.000s
I20240502 21:27:03.842499 5473 tablet_bootstrap.cc:492] T f61e34e65429408e83a13c7bc743e78a P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:27:03.848678 5461 raft_consensus.cc:348] T 4bc968ae4dcc4636a49257b6d5824bee P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } }
I20240502 21:27:03.848927 5461 raft_consensus.cc:729] T 4bc968ae4dcc4636a49257b6d5824bee P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: LEARNER
I20240502 21:27:03.849146 5461 consensus_queue.cc:260] T 4bc968ae4dcc4636a49257b6d5824bee P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } }
I20240502 21:27:03.853310 5461 ts_tablet_manager.cc:1398] T 4bc968ae4dcc4636a49257b6d5824bee P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20240502 21:27:03.854092 31556 tablet_copy_service.cc:342] P a44f41b0b6644fb6aa29476aed8c964f: Request end of tablet copy session 05f4fac41c4f4959b9b316cb112d0ada-4bc968ae4dcc4636a49257b6d5824bee received from {username='slave'} at 127.24.226.130:41329
I20240502 21:27:03.854243 31556 tablet_copy_service.cc:433] P a44f41b0b6644fb6aa29476aed8c964f: ending tablet copy session 05f4fac41c4f4959b9b316cb112d0ada-4bc968ae4dcc4636a49257b6d5824bee on tablet 4bc968ae4dcc4636a49257b6d5824bee with peer 05f4fac41c4f4959b9b316cb112d0ada
I20240502 21:27:03.854523 1945 tablet_service.cc:1467] Processing CreateTablet for tablet cbc9e365873143cc955d8160a24a19b8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dce95cf294b1486eaab9fd8fdf20de6d]), partition=
I20240502 21:27:03.854859 2208 tablet_service.cc:1467] Processing CreateTablet for tablet cbc9e365873143cc955d8160a24a19b8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dce95cf294b1486eaab9fd8fdf20de6d]), partition=
I20240502 21:27:03.855046 1814 tablet_service.cc:1467] Processing CreateTablet for tablet cbc9e365873143cc955d8160a24a19b8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dce95cf294b1486eaab9fd8fdf20de6d]), partition=
I20240502 21:27:03.855212 2208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbc9e365873143cc955d8160a24a19b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.855394 1945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbc9e365873143cc955d8160a24a19b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.855409 1814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbc9e365873143cc955d8160a24a19b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:03.863153 5462 tablet_bootstrap.cc:492] T cbc9e365873143cc955d8160a24a19b8 P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:27:03.864974 5462 tablet_bootstrap.cc:654] T cbc9e365873143cc955d8160a24a19b8 P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:03.873522 5470 tablet_bootstrap.cc:492] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:27:03.875398 5470 tablet_bootstrap.cc:654] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:03.878183 5536 ts_tablet_manager.cc:1888] T 654b4e1fe654455b855b7dae1203badf P 9dec645b473f40578e726f3e50e219bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:03.878336 5536 log.cc:1196] T 654b4e1fe654455b855b7dae1203badf P 9dec645b473f40578e726f3e50e219bf: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal/wals/654b4e1fe654455b855b7dae1203badf
I20240502 21:27:03.878839 5536 ts_tablet_manager.cc:1909] T 654b4e1fe654455b855b7dae1203badf P 9dec645b473f40578e726f3e50e219bf: Deleting consensus metadata
I20240502 21:27:03.883023 31379 catalog_manager.cc:4830] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): tablet 654b4e1fe654455b855b7dae1203badf (table rebalancer_dynamic_table_new_name [id=3880230f55a6456b8789cd24843fd807]) successfully deleted
I20240502 21:27:03.885120 31796 raft_consensus.cc:1203] T 1f743c7345184feaa50871dc49a1f136 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:03.885648 5545 tablet_bootstrap.cc:492] T cbc9e365873143cc955d8160a24a19b8 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:27:03.887202 5545 tablet_bootstrap.cc:654] T cbc9e365873143cc955d8160a24a19b8 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:03.910274 5462 tablet_bootstrap.cc:492] T cbc9e365873143cc955d8160a24a19b8 P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:27:03.910518 5462 ts_tablet_manager.cc:1367] T cbc9e365873143cc955d8160a24a19b8 P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.047s user 0.012s sys 0.000s
I20240502 21:27:03.911401 5462 raft_consensus.cc:348] T cbc9e365873143cc955d8160a24a19b8 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:03.913040 31925 raft_consensus.cc:1203] T 883c729aa26f47178f40a17ad3b7e590 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:03.913218 5462 raft_consensus.cc:374] T cbc9e365873143cc955d8160a24a19b8 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:03.913367 5462 raft_consensus.cc:729] T cbc9e365873143cc955d8160a24a19b8 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:27:03.916824 5545 tablet_bootstrap.cc:492] T cbc9e365873143cc955d8160a24a19b8 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:27:03.917070 5545 ts_tablet_manager.cc:1367] T cbc9e365873143cc955d8160a24a19b8 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20240502 21:27:03.917850 5545 raft_consensus.cc:348] T cbc9e365873143cc955d8160a24a19b8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:03.918756 5545 raft_consensus.cc:374] T cbc9e365873143cc955d8160a24a19b8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:03.918893 5545 raft_consensus.cc:729] T cbc9e365873143cc955d8160a24a19b8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:27:03.919173 5545 consensus_queue.cc:260] T cbc9e365873143cc955d8160a24a19b8 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:03.919829 5545 ts_tablet_manager.cc:1398] T cbc9e365873143cc955d8160a24a19b8 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:27:03.920193 5462 consensus_queue.cc:260] T cbc9e365873143cc955d8160a24a19b8 P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:03.925216 5462 ts_tablet_manager.cc:1398] T cbc9e365873143cc955d8160a24a19b8 P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20240502 21:27:03.926844 5470 tablet_bootstrap.cc:492] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:27:03.927098 5470 ts_tablet_manager.cc:1367] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.054s user 0.009s sys 0.003s
I20240502 21:27:03.927982 5470 raft_consensus.cc:348] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:03.928243 5470 raft_consensus.cc:374] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:03.928324 5470 raft_consensus.cc:729] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:27:03.928545 5470 consensus_queue.cc:260] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:03.929170 5470 ts_tablet_manager.cc:1398] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:03.931180 31665 raft_consensus.cc:1203] T b8db5de7df864ab983504e96fb738106 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:03.963778 5510 tablet_bootstrap.cc:492] T 381ac1d714204cef86c8e1a73e24b863 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:03.966028 5510 tablet_bootstrap.cc:492] T 381ac1d714204cef86c8e1a73e24b863 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap complete.
I20240502 21:27:03.966442 5510 ts_tablet_manager.cc:1367] T 381ac1d714204cef86c8e1a73e24b863 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.237s user 0.053s sys 0.010s
I20240502 21:27:03.971961 5510 raft_consensus.cc:348] T 381ac1d714204cef86c8e1a73e24b863 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } }
I20240502 21:27:03.972296 5510 raft_consensus.cc:729] T 381ac1d714204cef86c8e1a73e24b863 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: LEARNER
I20240502 21:27:03.972573 5510 consensus_queue.cc:260] T 381ac1d714204cef86c8e1a73e24b863 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } }
I20240502 21:27:03.976042 5510 ts_tablet_manager.cc:1398] T 381ac1d714204cef86c8e1a73e24b863 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.008s user 0.002s sys 0.001s
I20240502 21:27:03.986833 5473 tablet_bootstrap.cc:492] T f61e34e65429408e83a13c7bc743e78a P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:03.987207 5473 tablet_bootstrap.cc:492] T f61e34e65429408e83a13c7bc743e78a P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap complete.
I20240502 21:27:03.987457 5473 ts_tablet_manager.cc:1367] T f61e34e65429408e83a13c7bc743e78a P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.145s user 0.036s sys 0.009s
I20240502 21:27:03.988066 5473 raft_consensus.cc:348] T f61e34e65429408e83a13c7bc743e78a P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } }
I20240502 21:27:03.988246 5473 raft_consensus.cc:729] T f61e34e65429408e83a13c7bc743e78a P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: LEARNER
I20240502 21:27:03.988469 5473 consensus_queue.cc:260] T f61e34e65429408e83a13c7bc743e78a P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } }
I20240502 21:27:03.989151 5473 ts_tablet_manager.cc:1398] T f61e34e65429408e83a13c7bc743e78a P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 21:27:03.991016 31686 tablet_copy_service.cc:342] P 05f4fac41c4f4959b9b316cb112d0ada: Request end of tablet copy session 1ade0354c44c4c37ae480c7a94b11e1b-f61e34e65429408e83a13c7bc743e78a received from {username='slave'} at 127.24.226.135:46887
I20240502 21:27:03.991153 31686 tablet_copy_service.cc:433] P 05f4fac41c4f4959b9b316cb112d0ada: ending tablet copy session 1ade0354c44c4c37ae480c7a94b11e1b-f61e34e65429408e83a13c7bc743e78a on tablet f61e34e65429408e83a13c7bc743e78a with peer 1ade0354c44c4c37ae480c7a94b11e1b
I20240502 21:27:03.993866 31556 tablet_copy_service.cc:342] P a44f41b0b6644fb6aa29476aed8c964f: Request end of tablet copy session 1e04157b98dd4fad98858c290b73c3db-381ac1d714204cef86c8e1a73e24b863 received from {username='slave'} at 127.24.226.133:49939
I20240502 21:27:03.993965 31556 tablet_copy_service.cc:433] P a44f41b0b6644fb6aa29476aed8c964f: ending tablet copy session 1e04157b98dd4fad98858c290b73c3db-381ac1d714204cef86c8e1a73e24b863 on tablet 381ac1d714204cef86c8e1a73e24b863 with peer 1e04157b98dd4fad98858c290b73c3db
I20240502 21:27:03.994113 1832 raft_consensus.cc:1203] T 381ac1d714204cef86c8e1a73e24b863 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:04.041673 5215 raft_consensus.cc:482] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:04.041883 5215 raft_consensus.cc:504] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:04.042497 5215 leader_election.cc:288] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:27:04.043220 1965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbc9e365873143cc955d8160a24a19b8" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:27:04.043435 1965 raft_consensus.cc:2454] T cbc9e365873143cc955d8160a24a19b8 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 0.
I20240502 21:27:04.043767 1767 leader_election.cc:302] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:27:04.043771 2226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbc9e365873143cc955d8160a24a19b8" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:27:04.043993 2226 raft_consensus.cc:2454] T cbc9e365873143cc955d8160a24a19b8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 0.
I20240502 21:27:04.044024 5215 raft_consensus.cc:2781] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:04.044122 5215 raft_consensus.cc:482] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:04.044202 5215 raft_consensus.cc:3037] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:04.046694 5215 raft_consensus.cc:504] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:04.047261 5215 leader_election.cc:288] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Requested vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:27:04.048146 1965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbc9e365873143cc955d8160a24a19b8" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf"
I20240502 21:27:04.048319 1965 raft_consensus.cc:3037] T cbc9e365873143cc955d8160a24a19b8 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:04.049038 2226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbc9e365873143cc955d8160a24a19b8" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:27:04.049228 2226 raft_consensus.cc:3037] T cbc9e365873143cc955d8160a24a19b8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:04.050962 1965 raft_consensus.cc:2454] T cbc9e365873143cc955d8160a24a19b8 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:27:04.051088 2226 raft_consensus.cc:2454] T cbc9e365873143cc955d8160a24a19b8 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:27:04.051589 1769 leader_election.cc:302] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:27:04.052170 5215 raft_consensus.cc:2781] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:04.052708 5215 raft_consensus.cc:686] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Becoming Leader. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Running, Role: LEADER
I20240502 21:27:04.053069 5215 consensus_queue.cc:237] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:04.057319 31386 catalog_manager.cc:5484] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e04157b98dd4fad98858c290b73c3db (127.24.226.133). New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:04.070868 5202 consensus_queue.cc:1035] T f1815f008fc041adb2da8c1e626a354f P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:04.079308 5512 consensus_queue.cc:1035] T f1815f008fc041adb2da8c1e626a354f P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:04.095741 1833 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9bc57b8a0388416290997df544e5cc0b"
dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59794
I20240502 21:27:04.095921 1833 raft_consensus.cc:595] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Received request to transfer leadership
I20240502 21:27:04.123742 31536 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dd0198178da640688230aeb5a1106cb4"
dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50606
I20240502 21:27:04.123960 31536 raft_consensus.cc:595] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Received request to transfer leadership
I20240502 21:27:04.144333 31536 consensus_queue.cc:237] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:27:04.146190 1965 raft_consensus.cc:1261] T 3e6657723fe84e749dea768de44638e7 P 9dec645b473f40578e726f3e50e219bf [term 2 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 21:27:04.146510 2092 raft_consensus.cc:1261] T 3e6657723fe84e749dea768de44638e7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 2 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 21:27:04.146860 5531 consensus_queue.cc:1035] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:04.147143 5528 consensus_queue.cc:1035] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:04.151692 5530 raft_consensus.cc:2932] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:27:04.152213 2094 raft_consensus.cc:2932] T 3e6657723fe84e749dea768de44638e7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:27:04.152765 1965 raft_consensus.cc:2932] T 3e6657723fe84e749dea768de44638e7 P 9dec645b473f40578e726f3e50e219bf [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:27:04.155233 31382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3e6657723fe84e749dea768de44638e7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:04.156713 4801 consensus_queue.cc:1035] T 41029f1332c64ba58ee0ee45714ba2a8 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:04.158656 31386 catalog_manager.cc:5484] T 3e6657723fe84e749dea768de44638e7 P 9dec645b473f40578e726f3e50e219bf reported cstate change: config changed from index 3 to 5, VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) evicted. New cstate: current_term: 2 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:27:04.165743 1814 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e6657723fe84e749dea768de44638e7 with delete_type TABLET_DATA_TOMBSTONED (TS 1e04157b98dd4fad98858c290b73c3db not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47510
I20240502 21:27:04.172531 5621 tablet_replica.cc:330] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db: stopping tablet replica
I20240502 21:27:04.172998 5621 raft_consensus.cc:2229] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:04.173369 5621 raft_consensus.cc:2258] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:04.174391 5621 ts_tablet_manager.cc:1875] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:04.179220 5537 consensus_queue.cc:1035] T 41029f1332c64ba58ee0ee45714ba2a8 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:04.184324 5621 ts_tablet_manager.cc:1888] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 21:27:04.184556 5621 log.cc:1196] T 3e6657723fe84e749dea768de44638e7 P 1e04157b98dd4fad98858c290b73c3db: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal/wals/3e6657723fe84e749dea768de44638e7
I20240502 21:27:04.185658 31381 catalog_manager.cc:4830] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): tablet 3e6657723fe84e749dea768de44638e7 (table rebalancer_extra_table_18 [id=a35acd1fa3294955995e9c2c846a5de0]) successfully deleted
I20240502 21:27:04.197808 31392 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:04.204736 1965 raft_consensus.cc:1261] T cbc9e365873143cc955d8160a24a19b8 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:04.205989 2226 raft_consensus.cc:1261] T cbc9e365873143cc955d8160a24a19b8 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:04.206583 5215 consensus_queue.cc:1035] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:04.207289 5215 consensus_queue.cc:1035] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:04.246099 31392 catalog_manager.cc:5576] TS d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907) does not have the latest schema for tablet cbc9e365873143cc955d8160a24a19b8 (table rebalancer_dynamic_table [id=dce95cf294b1486eaab9fd8fdf20de6d]). Expected version 1 got 0
I20240502 21:27:04.250910 5531 consensus_queue.cc:1035] T 58be3e32928043e1af387efd99e43b63 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:04.262671 5531 consensus_queue.cc:1035] T 58be3e32928043e1af387efd99e43b63 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:04.263952 31392 catalog_manager.cc:5576] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753) does not have the latest schema for tablet cbc9e365873143cc955d8160a24a19b8 (table rebalancer_dynamic_table [id=dce95cf294b1486eaab9fd8fdf20de6d]). Expected version 1 got 0
I20240502 21:27:04.269996 5530 raft_consensus.cc:979] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f: : Instructing follower 08d7a0680d244b55b93d17f9ab57b84c to start an election
I20240502 21:27:04.270156 5530 raft_consensus.cc:1067] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Signalling peer 08d7a0680d244b55b93d17f9ab57b84c to start an election
I20240502 21:27:04.271078 31796 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dd0198178da640688230aeb5a1106cb4"
dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
from {username='slave'} at 127.24.226.129:44757
I20240502 21:27:04.271342 31796 raft_consensus.cc:482] T dd0198178da640688230aeb5a1106cb4 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:04.271463 31796 raft_consensus.cc:3037] T dd0198178da640688230aeb5a1106cb4 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:04.275490 31796 raft_consensus.cc:504] T dd0198178da640688230aeb5a1106cb4 P 08d7a0680d244b55b93d17f9ab57b84c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } }
I20240502 21:27:04.278334 5640 tablet.cc:1722] T cbc9e365873143cc955d8160a24a19b8 P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:04.279460 5640 tablet.cc:1620] T cbc9e365873143cc955d8160a24a19b8 P 9dec645b473f40578e726f3e50e219bf: MemRowSet was empty: no flush needed.
I20240502 21:27:04.280155 31533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd0198178da640688230aeb5a1106cb4" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:27:04.280346 31533 raft_consensus.cc:3032] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:04.280427 31533 raft_consensus.cc:729] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:27:04.280769 31533 consensus_queue.cc:260] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } }
I20240502 21:27:04.281160 31533 raft_consensus.cc:3037] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:04.285226 31533 raft_consensus.cc:2454] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 2.
I20240502 21:27:04.287133 2094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd0198178da640688230aeb5a1106cb4" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:27:04.287324 2094 raft_consensus.cc:3037] T dd0198178da640688230aeb5a1106cb4 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:04.287834 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd0198178da640688230aeb5a1106cb4" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:27:04.287997 31925 raft_consensus.cc:3037] T dd0198178da640688230aeb5a1106cb4 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:04.290040 2094 raft_consensus.cc:2454] T dd0198178da640688230aeb5a1106cb4 P 1ade0354c44c4c37ae480c7a94b11e1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 2.
I20240502 21:27:04.290655 31925 raft_consensus.cc:2454] T dd0198178da640688230aeb5a1106cb4 P 2d50f33557944bb480ea32199d9ee4f1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 2.
I20240502 21:27:04.291088 31732 leader_election.cc:302] T dd0198178da640688230aeb5a1106cb4 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 1ade0354c44c4c37ae480c7a94b11e1b, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:27:04.291373 4859 raft_consensus.cc:2781] T dd0198178da640688230aeb5a1106cb4 P 08d7a0680d244b55b93d17f9ab57b84c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:04.291518 4859 raft_consensus.cc:686] T dd0198178da640688230aeb5a1106cb4 P 08d7a0680d244b55b93d17f9ab57b84c [term 2 LEADER]: Becoming Leader. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Running, Role: LEADER
I20240502 21:27:04.291790 4859 consensus_queue.cc:237] T dd0198178da640688230aeb5a1106cb4 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } }
I20240502 21:27:04.296870 31386 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=dce95cf294b1486eaab9fd8fdf20de6d] alter complete (version 1)
I20240502 21:27:04.301033 31796 leader_election.cc:288] T dd0198178da640688230aeb5a1106cb4 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 2 election: Requested vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:27:04.313951 5635 tablet.cc:1722] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:04.314570 5635 tablet.cc:1620] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db: MemRowSet was empty: no flush needed.
I20240502 21:27:04.320616 31392 catalog_manager.cc:5484] T dd0198178da640688230aeb5a1106cb4 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: term changed from 1 to 2, leader changed from a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) to 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131). New cstate: current_term: 2 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:04.332880 5636 tablet.cc:1722] T cbc9e365873143cc955d8160a24a19b8 P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:04.335803 31392 catalog_manager.cc:5576] TS d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907) does not have the latest schema for tablet cbc9e365873143cc955d8160a24a19b8 (table rebalancer_dynamic_table [id=dce95cf294b1486eaab9fd8fdf20de6d]). Expected version 1 got 0
I20240502 21:27:04.335852 2094 raft_consensus.cc:1203] T f61e34e65429408e83a13c7bc743e78a P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:04.339561 5636 tablet.cc:1620] T cbc9e365873143cc955d8160a24a19b8 P d1406e92aec84ac585304e57bd37a40d: MemRowSet was empty: no flush needed.
I20240502 21:27:04.342003 1962 raft_consensus.cc:1203] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 21:27:04.346969 31392 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_extra_table_42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:04.360576 31516 tablet_service.cc:1467] Processing CreateTablet for tablet e84560e2ba25443cbba6babbbfcaf104 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=586f1e6f7d4740b2a48d4098618047a5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:04.361035 31516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e84560e2ba25443cbba6babbbfcaf104. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.362166 31775 tablet_service.cc:1467] Processing CreateTablet for tablet c8b447d3c1a24051a914b7ed642808fa (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=586f1e6f7d4740b2a48d4098618047a5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:04.362568 31775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8b447d3c1a24051a914b7ed642808fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.363200 2077 tablet_service.cc:1467] Processing CreateTablet for tablet c8b447d3c1a24051a914b7ed642808fa (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=586f1e6f7d4740b2a48d4098618047a5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:04.363605 2077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8b447d3c1a24051a914b7ed642808fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.364806 31906 tablet_service.cc:1467] Processing CreateTablet for tablet c8b447d3c1a24051a914b7ed642808fa (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=586f1e6f7d4740b2a48d4098618047a5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:04.365128 2208 tablet_service.cc:1467] Processing CreateTablet for tablet e84560e2ba25443cbba6babbbfcaf104 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=586f1e6f7d4740b2a48d4098618047a5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:04.365232 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8b447d3c1a24051a914b7ed642808fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.365530 2208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e84560e2ba25443cbba6babbbfcaf104. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.368856 1814 tablet_service.cc:1467] Processing CreateTablet for tablet e84560e2ba25443cbba6babbbfcaf104 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=586f1e6f7d4740b2a48d4098618047a5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:04.369303 1814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e84560e2ba25443cbba6babbbfcaf104. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.375803 31774 tablet_service.cc:1467] Processing CreateTablet for tablet 7cc4bbc0deb8464db065e23cda176437 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=586f1e6f7d4740b2a48d4098618047a5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:04.376226 31774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cc4bbc0deb8464db065e23cda176437. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.378729 5156 raft_consensus.cc:1050] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d: attempting to promote NON_VOTER 2d50f33557944bb480ea32199d9ee4f1 to VOTER
I20240502 21:27:04.379565 5156 consensus_queue.cc:237] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } }
I20240502 21:27:04.383709 31645 tablet_service.cc:1467] Processing CreateTablet for tablet 7cc4bbc0deb8464db065e23cda176437 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=586f1e6f7d4740b2a48d4098618047a5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:04.384110 31645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cc4bbc0deb8464db065e23cda176437. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.384727 31905 tablet_service.cc:1467] Processing CreateTablet for tablet 7cc4bbc0deb8464db065e23cda176437 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=586f1e6f7d4740b2a48d4098618047a5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:04.385116 31905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cc4bbc0deb8464db065e23cda176437. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.388545 5223 raft_consensus.cc:979] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db: : Instructing follower 08d7a0680d244b55b93d17f9ab57b84c to start an election
I20240502 21:27:04.388664 5223 raft_consensus.cc:1067] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Signalling peer 08d7a0680d244b55b93d17f9ab57b84c to start an election
I20240502 21:27:04.389732 31797 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9bc57b8a0388416290997df544e5cc0b"
dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
from {username='slave'} at 127.24.226.133:41593
I20240502 21:27:04.389930 31797 raft_consensus.cc:482] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:04.390038 31797 raft_consensus.cc:3037] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:04.390138 4559 raft_consensus.cc:1050] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c: attempting to promote NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada to VOTER
I20240502 21:27:04.391065 4559 consensus_queue.cc:237] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } }
I20240502 21:27:04.394984 31797 raft_consensus.cc:504] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } }
I20240502 21:27:04.396127 1965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bc57b8a0388416290997df544e5cc0b" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9dec645b473f40578e726f3e50e219bf"
I20240502 21:27:04.396354 1965 raft_consensus.cc:3037] T 9bc57b8a0388416290997df544e5cc0b P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:04.396660 1833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bc57b8a0388416290997df544e5cc0b" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:27:04.396885 1833 raft_consensus.cc:3032] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:04.397054 1833 raft_consensus.cc:729] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Running, Role: LEADER
I20240502 21:27:04.397367 1833 consensus_queue.cc:260] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } }
I20240502 21:27:04.397874 1833 raft_consensus.cc:3037] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:04.399271 1965 raft_consensus.cc:2454] T 9bc57b8a0388416290997df544e5cc0b P 9dec645b473f40578e726f3e50e219bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 2.
I20240502 21:27:04.399307 5652 tablet_bootstrap.cc:492] T c8b447d3c1a24051a914b7ed642808fa P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:27:04.400835 31797 leader_election.cc:288] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 2 election: Requested vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:27:04.401378 31667 raft_consensus.cc:1203] T 4bc968ae4dcc4636a49257b6d5824bee P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:04.402102 1833 raft_consensus.cc:2454] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 2.
I20240502 21:27:04.402117 1962 raft_consensus.cc:1226] T 9bc57b8a0388416290997df544e5cc0b P 9dec645b473f40578e726f3e50e219bf [term 2 FOLLOWER]: Rejecting Update request from peer 1e04157b98dd4fad98858c290b73c3db for earlier term 1. Current term is 2. Ops: []
I20240502 21:27:04.402491 31732 leader_election.cc:302] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 1e04157b98dd4fad98858c290b73c3db, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:27:04.402719 4859 raft_consensus.cc:2781] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:04.402842 4859 raft_consensus.cc:686] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c [term 2 LEADER]: Becoming Leader. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Running, Role: LEADER
I20240502 21:27:04.403067 4859 consensus_queue.cc:237] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } }
I20240502 21:27:04.406903 5545 tablet_bootstrap.cc:492] T e84560e2ba25443cbba6babbbfcaf104 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:27:04.408216 5652 tablet_bootstrap.cc:654] T c8b447d3c1a24051a914b7ed642808fa P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:04.409085 31389 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:04.412379 5470 tablet_bootstrap.cc:492] T e84560e2ba25443cbba6babbbfcaf104 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:27:04.413928 5470 tablet_bootstrap.cc:654] T e84560e2ba25443cbba6babbbfcaf104 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:04.417371 5530 raft_consensus.cc:1050] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f: attempting to promote NON_VOTER 1e04157b98dd4fad98858c290b73c3db to VOTER
I20240502 21:27:04.418223 5530 consensus_queue.cc:237] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } }
I20240502 21:27:04.418857 2094 raft_consensus.cc:1261] T 883c729aa26f47178f40a17ad3b7e590 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:04.419512 5522 consensus_queue.cc:1035] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:04.425233 5636 tablet.cc:1722] T cbc9e365873143cc955d8160a24a19b8 P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:04.425695 5636 tablet.cc:1620] T cbc9e365873143cc955d8160a24a19b8 P d1406e92aec84ac585304e57bd37a40d: MemRowSet was empty: no flush needed.
I20240502 21:27:04.426620 5202 raft_consensus.cc:1050] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1: attempting to promote NON_VOTER 08d7a0680d244b55b93d17f9ab57b84c to VOTER
I20240502 21:27:04.427081 31389 catalog_manager.cc:5484] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: term changed from 1 to 2, leader changed from 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) to 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131). New cstate: current_term: 2 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:04.427295 5202 consensus_queue.cc:237] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } }
I20240502 21:27:04.439232 31925 raft_consensus.cc:1261] T 883c729aa26f47178f40a17ad3b7e590 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEARNER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:04.439308 1832 raft_consensus.cc:1261] T 883c729aa26f47178f40a17ad3b7e590 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:04.439875 5522 consensus_queue.cc:1035] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:04.440387 5522 consensus_queue.cc:1035] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:04.440908 5545 tablet_bootstrap.cc:654] T e84560e2ba25443cbba6babbbfcaf104 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:04.442589 5640 tablet.cc:1722] T cbc9e365873143cc955d8160a24a19b8 P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:04.443154 5640 tablet.cc:1620] T cbc9e365873143cc955d8160a24a19b8 P 9dec645b473f40578e726f3e50e219bf: MemRowSet was empty: no flush needed.
I20240502 21:27:04.460559 1962 raft_consensus.cc:1261] T b8db5de7df864ab983504e96fb738106 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:04.460608 31667 raft_consensus.cc:1261] T b8db5de7df864ab983504e96fb738106 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:04.461042 1833 raft_consensus.cc:1261] T b8db5de7df864ab983504e96fb738106 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:04.461256 4859 consensus_queue.cc:1035] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:04.461553 5417 consensus_queue.cc:1035] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:04.461802 5545 tablet_bootstrap.cc:492] T e84560e2ba25443cbba6babbbfcaf104 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:27:04.462002 2092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bc57b8a0388416290997df544e5cc0b" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:27:04.462078 5417 consensus_queue.cc:1035] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:04.462105 5545 ts_tablet_manager.cc:1367] T e84560e2ba25443cbba6babbbfcaf104 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.055s user 0.008s sys 0.005s
I20240502 21:27:04.462208 2092 raft_consensus.cc:3037] T 9bc57b8a0388416290997df544e5cc0b P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:04.463119 5545 raft_consensus.cc:348] T e84560e2ba25443cbba6babbbfcaf104 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:04.463678 5649 tablet_bootstrap.cc:492] T c8b447d3c1a24051a914b7ed642808fa P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:27:04.464515 5545 raft_consensus.cc:374] T e84560e2ba25443cbba6babbbfcaf104 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:04.464686 5545 raft_consensus.cc:729] T e84560e2ba25443cbba6babbbfcaf104 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:27:04.465528 5545 consensus_queue.cc:260] T e84560e2ba25443cbba6babbbfcaf104 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:04.466418 5649 tablet_bootstrap.cc:654] T c8b447d3c1a24051a914b7ed642808fa P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:04.466516 2092 raft_consensus.cc:2454] T 9bc57b8a0388416290997df544e5cc0b P 1ade0354c44c4c37ae480c7a94b11e1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 2.
I20240502 21:27:04.467020 5545 ts_tablet_manager.cc:1398] T e84560e2ba25443cbba6babbbfcaf104 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 21:27:04.471199 5635 tablet.cc:1722] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:04.471930 5635 tablet.cc:1620] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db: MemRowSet was empty: no flush needed.
I20240502 21:27:04.474344 31389 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=dce95cf294b1486eaab9fd8fdf20de6d] alter complete (version 2)
I20240502 21:27:04.475703 5656 tablet_bootstrap.cc:492] T 7cc4bbc0deb8464db065e23cda176437 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:27:04.476495 5649 tablet_bootstrap.cc:492] T c8b447d3c1a24051a914b7ed642808fa P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:27:04.476684 5649 ts_tablet_manager.cc:1367] T c8b447d3c1a24051a914b7ed642808fa P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20240502 21:27:04.477398 5649 raft_consensus.cc:348] T c8b447d3c1a24051a914b7ed642808fa P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:04.477636 5649 raft_consensus.cc:374] T c8b447d3c1a24051a914b7ed642808fa P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:04.477728 5649 raft_consensus.cc:729] T c8b447d3c1a24051a914b7ed642808fa P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:27:04.478018 5649 consensus_queue.cc:260] T c8b447d3c1a24051a914b7ed642808fa P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:04.480885 5649 ts_tablet_manager.cc:1398] T c8b447d3c1a24051a914b7ed642808fa P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 21:27:04.481177 5649 tablet_bootstrap.cc:492] T 7cc4bbc0deb8464db065e23cda176437 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:27:04.483080 31797 raft_consensus.cc:1261] T 1f743c7345184feaa50871dc49a1f136 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEARNER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:04.483608 5526 consensus_queue.cc:1035] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:04.484313 5649 tablet_bootstrap.cc:654] T 7cc4bbc0deb8464db065e23cda176437 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:04.488813 5156 raft_consensus.cc:2932] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:27:04.493098 5526 raft_consensus.cc:482] T c8b447d3c1a24051a914b7ed642808fa P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:04.493237 5526 raft_consensus.cc:504] T c8b447d3c1a24051a914b7ed642808fa P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:04.494035 5526 leader_election.cc:288] T c8b447d3c1a24051a914b7ed642808fa P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:27:04.494810 31796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b447d3c1a24051a914b7ed642808fa" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:27:04.495167 5649 tablet_bootstrap.cc:492] T 7cc4bbc0deb8464db065e23cda176437 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:27:04.495334 5649 ts_tablet_manager.cc:1367] T 7cc4bbc0deb8464db065e23cda176437 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.003s
W20240502 21:27:04.495640 31862 leader_election.cc:341] T c8b447d3c1a24051a914b7ed642808fa P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:04.496107 5649 raft_consensus.cc:348] T 7cc4bbc0deb8464db065e23cda176437 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:04.496315 5649 raft_consensus.cc:374] T 7cc4bbc0deb8464db065e23cda176437 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:04.496397 5649 raft_consensus.cc:729] T 7cc4bbc0deb8464db065e23cda176437 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:27:04.496098 2092 raft_consensus.cc:2932] T 883c729aa26f47178f40a17ad3b7e590 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:27:04.496685 5649 consensus_queue.cc:260] T 7cc4bbc0deb8464db065e23cda176437 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
W20240502 21:27:04.497426 2164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.226.132:35933, user_credentials={real_user=slave}} blocked reactor thread for 57580.3us
W20240502 21:27:04.498975 1770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.226.130:37645, user_credentials={real_user=slave}} blocked reactor thread for 105914us
I20240502 21:27:04.498373 5649 ts_tablet_manager.cc:1398] T 7cc4bbc0deb8464db065e23cda176437 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:27:04.502924 5417 raft_consensus.cc:2932] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:27:04.504721 2092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b447d3c1a24051a914b7ed642808fa" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
W20240502 21:27:04.505167 31862 leader_election.cc:341] T c8b447d3c1a24051a914b7ed642808fa P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:04.505281 31862 leader_election.cc:302] T c8b447d3c1a24051a914b7ed642808fa P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1; no voters: 08d7a0680d244b55b93d17f9ab57b84c, 1ade0354c44c4c37ae480c7a94b11e1b
I20240502 21:27:04.505474 5526 raft_consensus.cc:2726] T c8b447d3c1a24051a914b7ed642808fa P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:27:04.506009 31533 raft_consensus.cc:1261] T 1f743c7345184feaa50871dc49a1f136 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:04.506140 2226 raft_consensus.cc:1261] T 1f743c7345184feaa50871dc49a1f136 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:04.506736 5512 consensus_queue.cc:1035] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:04.507166 5512 consensus_queue.cc:1035] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:04.507436 5656 tablet_bootstrap.cc:654] T 7cc4bbc0deb8464db065e23cda176437 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:04.513206 5652 tablet_bootstrap.cc:492] T c8b447d3c1a24051a914b7ed642808fa P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:27:04.513379 5652 ts_tablet_manager.cc:1367] T c8b447d3c1a24051a914b7ed642808fa P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.114s user 0.002s sys 0.007s
I20240502 21:27:04.514102 31386 catalog_manager.cc:5484] T 883c729aa26f47178f40a17ad3b7e590 P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: config changed from index 2 to 3, 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:27:04.517450 1832 raft_consensus.cc:2932] T b8db5de7df864ab983504e96fb738106 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:27:04.514415 5652 raft_consensus.cc:348] T c8b447d3c1a24051a914b7ed642808fa P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:04.520246 5652 raft_consensus.cc:374] T c8b447d3c1a24051a914b7ed642808fa P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:04.520530 5652 raft_consensus.cc:729] T c8b447d3c1a24051a914b7ed642808fa P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:27:04.520990 5652 consensus_queue.cc:260] T c8b447d3c1a24051a914b7ed642808fa P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:04.526074 5656 tablet_bootstrap.cc:492] T 7cc4bbc0deb8464db065e23cda176437 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:27:04.526283 5656 ts_tablet_manager.cc:1367] T 7cc4bbc0deb8464db065e23cda176437 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.051s user 0.005s sys 0.000s
I20240502 21:27:04.526903 5656 raft_consensus.cc:348] T 7cc4bbc0deb8464db065e23cda176437 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:04.527166 5656 raft_consensus.cc:374] T 7cc4bbc0deb8464db065e23cda176437 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:04.527266 5656 raft_consensus.cc:729] T 7cc4bbc0deb8464db065e23cda176437 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:27:04.527498 5656 consensus_queue.cc:260] T 7cc4bbc0deb8464db065e23cda176437 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:04.527972 31796 raft_consensus.cc:1261] T 381ac1d714204cef86c8e1a73e24b863 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:04.528576 5528 consensus_queue.cc:1035] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:04.529177 5656 ts_tablet_manager.cc:1398] T 7cc4bbc0deb8464db065e23cda176437 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:27:04.533629 31925 raft_consensus.cc:2932] T 883c729aa26f47178f40a17ad3b7e590 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:27:04.536767 1833 raft_consensus.cc:1261] T 381ac1d714204cef86c8e1a73e24b863 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:04.537242 5528 consensus_queue.cc:1035] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:04.531292 31389 catalog_manager.cc:5484] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:04.541324 5652 ts_tablet_manager.cc:1398] T c8b447d3c1a24051a914b7ed642808fa P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.027s user 0.000s sys 0.001s
I20240502 21:27:04.543778 5512 raft_consensus.cc:2932] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:27:04.544303 2226 raft_consensus.cc:2932] T 1f743c7345184feaa50871dc49a1f136 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:27:04.548386 5470 tablet_bootstrap.cc:492] T e84560e2ba25443cbba6babbbfcaf104 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:27:04.548548 5470 ts_tablet_manager.cc:1367] T e84560e2ba25443cbba6babbbfcaf104 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.136s user 0.006s sys 0.006s
I20240502 21:27:04.549365 5470 raft_consensus.cc:348] T e84560e2ba25443cbba6babbbfcaf104 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:04.549599 5470 raft_consensus.cc:374] T e84560e2ba25443cbba6babbbfcaf104 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:04.549692 5470 raft_consensus.cc:729] T e84560e2ba25443cbba6babbbfcaf104 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:27:04.549891 5470 consensus_queue.cc:260] T e84560e2ba25443cbba6babbbfcaf104 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:04.551666 5470 ts_tablet_manager.cc:1398] T e84560e2ba25443cbba6babbbfcaf104 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:27:04.552040 31667 raft_consensus.cc:2932] T b8db5de7df864ab983504e96fb738106 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:27:04.553418 31797 raft_consensus.cc:2932] T 1f743c7345184feaa50871dc49a1f136 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:27:04.555622 31925 raft_consensus.cc:1261] T 381ac1d714204cef86c8e1a73e24b863 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:04.556241 5530 consensus_queue.cc:1035] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:04.557799 5647 tablet_bootstrap.cc:492] T c8b447d3c1a24051a914b7ed642808fa P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:27:04.560009 5647 tablet_bootstrap.cc:654] T c8b447d3c1a24051a914b7ed642808fa P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:04.561401 5650 tablet_bootstrap.cc:492] T e84560e2ba25443cbba6babbbfcaf104 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:27:04.561177 31386 catalog_manager.cc:5484] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: config changed from index 2 to 3, 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:04.562976 5650 tablet_bootstrap.cc:654] T e84560e2ba25443cbba6babbbfcaf104 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:04.564110 31536 raft_consensus.cc:2932] T 1f743c7345184feaa50871dc49a1f136 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:27:04.566919 1834 raft_consensus.cc:2932] T 883c729aa26f47178f40a17ad3b7e590 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:27:04.581043 31393 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:04.581437 31923 consensus_queue.cc:237] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } }
I20240502 21:27:04.582808 5528 raft_consensus.cc:2932] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:27:04.586367 2225 raft_consensus.cc:1261] T 1f743c7345184feaa50871dc49a1f136 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:04.587355 5202 consensus_queue.cc:1035] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:04.587291 1965 raft_consensus.cc:2932] T b8db5de7df864ab983504e96fb738106 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:27:04.589449 31795 consensus_queue.cc:237] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } }
I20240502 21:27:04.589382 31793 raft_consensus.cc:2932] T 381ac1d714204cef86c8e1a73e24b863 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:27:04.595248 1832 raft_consensus.cc:1261] T b8db5de7df864ab983504e96fb738106 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:04.595779 5677 consensus_queue.cc:1035] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:04.596237 5202 raft_consensus.cc:2932] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:27:04.596803 31797 raft_consensus.cc:1261] T 1f743c7345184feaa50871dc49a1f136 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:04.597496 5512 consensus_queue.cc:1035] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:04.597815 4859 raft_consensus.cc:2932] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:27:04.598371 2225 raft_consensus.cc:2932] T 1f743c7345184feaa50871dc49a1f136 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:27:04.602495 1833 raft_consensus.cc:2932] T 381ac1d714204cef86c8e1a73e24b863 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:27:04.604044 31388 catalog_manager.cc:5484] T 381ac1d714204cef86c8e1a73e24b863 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index 4 to 5, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:27:04.608896 31382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1f743c7345184feaa50871dc49a1f136 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:04.609886 1834 raft_consensus.cc:2932] T b8db5de7df864ab983504e96fb738106 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:27:04.610585 5640 tablet.cc:1722] T cbc9e365873143cc955d8160a24a19b8 P 9dec645b473f40578e726f3e50e219bf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:04.611264 31388 catalog_manager.cc:5576] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863) does not have the latest schema for tablet cbc9e365873143cc955d8160a24a19b8 (table rebalancer_dynamic_table_new_name [id=dce95cf294b1486eaab9fd8fdf20de6d]). Expected version 3 got 2
I20240502 21:27:04.612025 5635 tablet.cc:1722] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:04.614948 31388 catalog_manager.cc:5484] T 1f743c7345184feaa50871dc49a1f136 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: config changed from index 3 to 4, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:04.617583 31923 raft_consensus.cc:2932] T 381ac1d714204cef86c8e1a73e24b863 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:27:04.620877 31796 raft_consensus.cc:2932] T 1f743c7345184feaa50871dc49a1f136 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:27:04.623936 31665 raft_consensus.cc:1261] T b8db5de7df864ab983504e96fb738106 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:04.624934 5678 consensus_queue.cc:1035] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:04.626863 31665 raft_consensus.cc:2932] T b8db5de7df864ab983504e96fb738106 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:27:04.627194 5636 tablet.cc:1722] T cbc9e365873143cc955d8160a24a19b8 P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:04.616546 31381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b8db5de7df864ab983504e96fb738106 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:04.618121 31393 catalog_manager.cc:5484] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index 3 to 4, VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) evicted. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:04.636302 5650 tablet_bootstrap.cc:492] T e84560e2ba25443cbba6babbbfcaf104 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:27:04.636624 5650 ts_tablet_manager.cc:1367] T e84560e2ba25443cbba6babbbfcaf104 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.075s user 0.003s sys 0.000s
I20240502 21:27:04.639545 1945 tablet_service.cc:1514] Processing DeleteTablet for tablet b8db5de7df864ab983504e96fb738106 with delete_type TABLET_DATA_TOMBSTONED (TS 9dec645b473f40578e726f3e50e219bf not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33002
I20240502 21:27:04.641326 5650 raft_consensus.cc:348] T e84560e2ba25443cbba6babbbfcaf104 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:04.641675 5650 raft_consensus.cc:374] T e84560e2ba25443cbba6babbbfcaf104 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:04.643040 31389 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table_new_name [id=dce95cf294b1486eaab9fd8fdf20de6d] alter complete (version 3)
I20240502 21:27:04.641813 5650 raft_consensus.cc:729] T e84560e2ba25443cbba6babbbfcaf104 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:27:04.644523 5650 consensus_queue.cc:260] T e84560e2ba25443cbba6babbbfcaf104 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:04.651337 5687 tablet_replica.cc:330] T b8db5de7df864ab983504e96fb738106 P 9dec645b473f40578e726f3e50e219bf: stopping tablet replica
I20240502 21:27:04.651634 5687 raft_consensus.cc:2229] T b8db5de7df864ab983504e96fb738106 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:04.652056 5650 ts_tablet_manager.cc:1398] T e84560e2ba25443cbba6babbbfcaf104 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 21:27:04.653280 5687 raft_consensus.cc:2258] T b8db5de7df864ab983504e96fb738106 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:04.655931 5687 ts_tablet_manager.cc:1875] T b8db5de7df864ab983504e96fb738106 P 9dec645b473f40578e726f3e50e219bf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:04.654920 31516 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f743c7345184feaa50871dc49a1f136 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 3 (current committed config index is 4)) from {username='slave'} at 127.0.0.1:45194
I20240502 21:27:04.658887 31533 consensus_queue.cc:237] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } }
I20240502 21:27:04.662132 31923 raft_consensus.cc:1261] T 381ac1d714204cef86c8e1a73e24b863 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:04.662634 5530 consensus_queue.cc:1035] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:04.664597 5647 tablet_bootstrap.cc:492] T c8b447d3c1a24051a914b7ed642808fa P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:27:04.665153 5528 raft_consensus.cc:2932] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:27:04.665830 5647 ts_tablet_manager.cc:1367] T c8b447d3c1a24051a914b7ed642808fa P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.108s user 0.004s sys 0.000s
I20240502 21:27:04.666575 5647 raft_consensus.cc:348] T c8b447d3c1a24051a914b7ed642808fa P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:04.666671 5645 raft_consensus.cc:482] T c8b447d3c1a24051a914b7ed642808fa P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:04.666858 5645 raft_consensus.cc:504] T c8b447d3c1a24051a914b7ed642808fa P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:04.667659 5645 leader_election.cc:288] T c8b447d3c1a24051a914b7ed642808fa P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:27:04.667807 5647 raft_consensus.cc:374] T c8b447d3c1a24051a914b7ed642808fa P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:04.667934 5647 raft_consensus.cc:729] T c8b447d3c1a24051a914b7ed642808fa P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:27:04.668252 5647 consensus_queue.cc:260] T c8b447d3c1a24051a914b7ed642808fa P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:04.668584 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b447d3c1a24051a914b7ed642808fa" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:27:04.668633 1834 raft_consensus.cc:1261] T 381ac1d714204cef86c8e1a73e24b863 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:04.668859 31925 raft_consensus.cc:2454] T c8b447d3c1a24051a914b7ed642808fa P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 0.
I20240502 21:27:04.670222 2033 leader_election.cc:302] T c8b447d3c1a24051a914b7ed642808fa P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:27:04.670779 5647 ts_tablet_manager.cc:1398] T c8b447d3c1a24051a914b7ed642808fa P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 21:27:04.670948 5645 raft_consensus.cc:2781] T c8b447d3c1a24051a914b7ed642808fa P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:04.671098 5645 raft_consensus.cc:482] T c8b447d3c1a24051a914b7ed642808fa P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:04.671191 5645 raft_consensus.cc:3037] T c8b447d3c1a24051a914b7ed642808fa P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:04.672334 5647 tablet_bootstrap.cc:492] T 7cc4bbc0deb8464db065e23cda176437 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:27:04.672334 31796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b447d3c1a24051a914b7ed642808fa" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:27:04.672643 31796 raft_consensus.cc:2454] T c8b447d3c1a24051a914b7ed642808fa P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 0.
I20240502 21:27:04.673478 5645 raft_consensus.cc:504] T c8b447d3c1a24051a914b7ed642808fa P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:04.673859 5647 tablet_bootstrap.cc:654] T 7cc4bbc0deb8464db065e23cda176437 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:04.674811 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b447d3c1a24051a914b7ed642808fa" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:27:04.676151 31925 raft_consensus.cc:3037] T c8b447d3c1a24051a914b7ed642808fa P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:04.676226 5645 leader_election.cc:288] T c8b447d3c1a24051a914b7ed642808fa P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 election: Requested vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:27:04.676913 31382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 381ac1d714204cef86c8e1a73e24b863 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:04.677984 31796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b447d3c1a24051a914b7ed642808fa" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:27:04.678215 31796 raft_consensus.cc:3037] T c8b447d3c1a24051a914b7ed642808fa P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:04.678689 31925 raft_consensus.cc:2454] T c8b447d3c1a24051a914b7ed642808fa P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 1.
I20240502 21:27:04.679096 2033 leader_election.cc:302] T c8b447d3c1a24051a914b7ed642808fa P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:27:04.679474 5645 raft_consensus.cc:2781] T c8b447d3c1a24051a914b7ed642808fa P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:04.679656 5645 raft_consensus.cc:686] T c8b447d3c1a24051a914b7ed642808fa P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEADER]: Becoming Leader. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Running, Role: LEADER
I20240502 21:27:04.679989 5645 consensus_queue.cc:237] T c8b447d3c1a24051a914b7ed642808fa P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:04.680251 5687 ts_tablet_manager.cc:1888] T b8db5de7df864ab983504e96fb738106 P 9dec645b473f40578e726f3e50e219bf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:27:04.680650 5687 log.cc:1196] T b8db5de7df864ab983504e96fb738106 P 9dec645b473f40578e726f3e50e219bf: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal/wals/b8db5de7df864ab983504e96fb738106
I20240502 21:27:04.681389 31379 catalog_manager.cc:4830] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): tablet b8db5de7df864ab983504e96fb738106 (table rebalancer_extra_table_24 [id=da186563aba24f6d860086d2a27eeff3]) successfully deleted
I20240502 21:27:04.684842 31796 raft_consensus.cc:2454] T c8b447d3c1a24051a914b7ed642808fa P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 1.
I20240502 21:27:04.687062 31533 raft_consensus.cc:1261] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [term 2 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:27:04.687568 5528 consensus_queue.cc:1035] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.010s
I20240502 21:27:04.687441 31923 raft_consensus.cc:2932] T 381ac1d714204cef86c8e1a73e24b863 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:27:04.688288 31393 catalog_manager.cc:5484] T 381ac1d714204cef86c8e1a73e24b863 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: config changed from index 5 to 6, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:04.689168 1834 raft_consensus.cc:2932] T 381ac1d714204cef86c8e1a73e24b863 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:27:04.693118 5677 consensus_queue.cc:1035] T dd0198178da640688230aeb5a1106cb4 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:04.697168 5512 raft_consensus.cc:482] T 7cc4bbc0deb8464db065e23cda176437 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:04.697315 5512 raft_consensus.cc:504] T 7cc4bbc0deb8464db065e23cda176437 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:04.698050 5512 leader_election.cc:288] T 7cc4bbc0deb8464db065e23cda176437 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:27:04.700076 5647 tablet_bootstrap.cc:492] T 7cc4bbc0deb8464db065e23cda176437 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:27:04.700237 5647 ts_tablet_manager.cc:1367] T 7cc4bbc0deb8464db065e23cda176437 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.005s
I20240502 21:27:04.700515 31667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc4bbc0deb8464db065e23cda176437" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:27:04.700750 31667 raft_consensus.cc:2454] T 7cc4bbc0deb8464db065e23cda176437 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:27:04.700910 5647 raft_consensus.cc:348] T 7cc4bbc0deb8464db065e23cda176437 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:04.701076 5647 raft_consensus.cc:374] T 7cc4bbc0deb8464db065e23cda176437 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:04.701144 5647 raft_consensus.cc:729] T 7cc4bbc0deb8464db065e23cda176437 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:27:04.701217 31863 leader_election.cc:302] T 7cc4bbc0deb8464db065e23cda176437 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:27:04.701372 5647 consensus_queue.cc:260] T 7cc4bbc0deb8464db065e23cda176437 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:04.701494 5512 raft_consensus.cc:2781] T 7cc4bbc0deb8464db065e23cda176437 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:04.701573 5512 raft_consensus.cc:482] T 7cc4bbc0deb8464db065e23cda176437 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:04.701632 5512 raft_consensus.cc:3037] T 7cc4bbc0deb8464db065e23cda176437 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:04.702749 31775 tablet_service.cc:1514] Processing DeleteTablet for tablet 381ac1d714204cef86c8e1a73e24b863 with delete_type TABLET_DATA_TOMBSTONED (TS 08d7a0680d244b55b93d17f9ab57b84c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51596
I20240502 21:27:04.703907 5512 raft_consensus.cc:504] T 7cc4bbc0deb8464db065e23cda176437 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:04.704797 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc4bbc0deb8464db065e23cda176437" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:27:04.704996 31797 raft_consensus.cc:3037] T 7cc4bbc0deb8464db065e23cda176437 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:04.707561 31797 raft_consensus.cc:2454] T 7cc4bbc0deb8464db065e23cda176437 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:27:04.708631 31862 leader_election.cc:302] T 7cc4bbc0deb8464db065e23cda176437 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:27:04.708937 5202 raft_consensus.cc:2781] T 7cc4bbc0deb8464db065e23cda176437 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:04.709036 5202 raft_consensus.cc:686] T 7cc4bbc0deb8464db065e23cda176437 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Running, Role: LEADER
I20240502 21:27:04.709298 5202 consensus_queue.cc:237] T 7cc4bbc0deb8464db065e23cda176437 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:04.711287 5647 ts_tablet_manager.cc:1398] T 7cc4bbc0deb8464db065e23cda176437 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 21:27:04.712567 5688 tablet_replica.cc:330] T 1f743c7345184feaa50871dc49a1f136 P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:27:04.712682 31667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc4bbc0deb8464db065e23cda176437" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:27:04.712786 5688 raft_consensus.cc:2229] T 1f743c7345184feaa50871dc49a1f136 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:04.712812 31667 raft_consensus.cc:3037] T 7cc4bbc0deb8464db065e23cda176437 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:04.712976 5688 raft_consensus.cc:2258] T 1f743c7345184feaa50871dc49a1f136 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:04.714283 5653 raft_consensus.cc:482] T e84560e2ba25443cbba6babbbfcaf104 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:04.714416 5653 raft_consensus.cc:504] T e84560e2ba25443cbba6babbbfcaf104 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:04.715057 5653 leader_election.cc:288] T e84560e2ba25443cbba6babbbfcaf104 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:27:04.715342 31667 raft_consensus.cc:2454] T 7cc4bbc0deb8464db065e23cda176437 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:27:04.715719 2226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84560e2ba25443cbba6babbbfcaf104" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:27:04.715862 5512 leader_election.cc:288] T 7cc4bbc0deb8464db065e23cda176437 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Requested vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:27:04.715986 2226 raft_consensus.cc:2454] T e84560e2ba25443cbba6babbbfcaf104 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 0.
I20240502 21:27:04.716526 1769 leader_election.cc:302] T e84560e2ba25443cbba6babbbfcaf104 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:27:04.717296 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84560e2ba25443cbba6babbbfcaf104" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:27:04.717554 31536 raft_consensus.cc:2454] T e84560e2ba25443cbba6babbbfcaf104 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 0.
I20240502 21:27:04.718267 5653 raft_consensus.cc:2781] T e84560e2ba25443cbba6babbbfcaf104 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:04.718395 5653 raft_consensus.cc:482] T e84560e2ba25443cbba6babbbfcaf104 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:04.718502 5653 raft_consensus.cc:3037] T e84560e2ba25443cbba6babbbfcaf104 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:04.719596 5692 tablet_replica.cc:330] T 381ac1d714204cef86c8e1a73e24b863 P 08d7a0680d244b55b93d17f9ab57b84c: stopping tablet replica
I20240502 21:27:04.719846 5692 raft_consensus.cc:2229] T 381ac1d714204cef86c8e1a73e24b863 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:04.720062 5692 raft_consensus.cc:2258] T 381ac1d714204cef86c8e1a73e24b863 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:04.720973 5692 ts_tablet_manager.cc:1875] T 381ac1d714204cef86c8e1a73e24b863 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:04.721215 5653 raft_consensus.cc:504] T e84560e2ba25443cbba6babbbfcaf104 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:04.722481 31796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc4bbc0deb8464db065e23cda176437" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:27:04.722944 31796 raft_consensus.cc:2362] T 7cc4bbc0deb8464db065e23cda176437 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1. Re-sending same reply.
I20240502 21:27:04.723639 5089 raft_consensus.cc:1050] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f: attempting to promote NON_VOTER 9dec645b473f40578e726f3e50e219bf to VOTER
I20240502 21:27:04.724529 5089 consensus_queue.cc:237] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:27:04.726040 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84560e2ba25443cbba6babbbfcaf104" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:27:04.726240 5692 ts_tablet_manager.cc:1888] T 381ac1d714204cef86c8e1a73e24b863 P 08d7a0680d244b55b93d17f9ab57b84c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:04.726230 31536 raft_consensus.cc:3037] T e84560e2ba25443cbba6babbbfcaf104 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:04.726334 5692 log.cc:1196] T 381ac1d714204cef86c8e1a73e24b863 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/wals/381ac1d714204cef86c8e1a73e24b863
I20240502 21:27:04.728948 5653 leader_election.cc:288] T e84560e2ba25443cbba6babbbfcaf104 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Requested vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:27:04.729168 2226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84560e2ba25443cbba6babbbfcaf104" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:27:04.729351 2226 raft_consensus.cc:3037] T e84560e2ba25443cbba6babbbfcaf104 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:04.729362 31381 catalog_manager.cc:4830] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): tablet 381ac1d714204cef86c8e1a73e24b863 (table rebalance_test_table_19 [id=654ed3d4f1ef4492ac274c10f2ba1f3d]) successfully deleted
I20240502 21:27:04.731484 31536 raft_consensus.cc:2454] T e84560e2ba25443cbba6babbbfcaf104 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:27:04.731741 2226 raft_consensus.cc:2454] T e84560e2ba25443cbba6babbbfcaf104 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:27:04.731909 1770 leader_election.cc:302] T e84560e2ba25443cbba6babbbfcaf104 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:27:04.732372 5653 raft_consensus.cc:2781] T e84560e2ba25443cbba6babbbfcaf104 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:04.732558 5653 raft_consensus.cc:686] T e84560e2ba25443cbba6babbbfcaf104 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Becoming Leader. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Running, Role: LEADER
I20240502 21:27:04.732523 31388 catalog_manager.cc:5484] T 7cc4bbc0deb8464db065e23cda176437 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132). New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:04.732816 5653 consensus_queue.cc:237] T e84560e2ba25443cbba6babbbfcaf104 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:04.737838 31393 catalog_manager.cc:5484] T e84560e2ba25443cbba6babbbfcaf104 P 1e04157b98dd4fad98858c290b73c3db reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e04157b98dd4fad98858c290b73c3db (127.24.226.133). New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:04.746372 31393 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:04.746552 31393 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:04.749755 31393 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet cbc9e365873143cc955d8160a24a19b8
I20240502 21:27:04.750828 2208 tablet_service.cc:1514] Processing DeleteTablet for tablet cbc9e365873143cc955d8160a24a19b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:04 UTC) from {username='slave'} at 127.0.0.1:36984
I20240502 21:27:04.750864 1814 tablet_service.cc:1514] Processing DeleteTablet for tablet cbc9e365873143cc955d8160a24a19b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:04 UTC) from {username='slave'} at 127.0.0.1:47510
I20240502 21:27:04.751765 1945 tablet_service.cc:1514] Processing DeleteTablet for tablet cbc9e365873143cc955d8160a24a19b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:04 UTC) from {username='slave'} at 127.0.0.1:33002
I20240502 21:27:04.752030 5687 tablet_replica.cc:330] T cbc9e365873143cc955d8160a24a19b8 P 9dec645b473f40578e726f3e50e219bf: stopping tablet replica
I20240502 21:27:04.752283 5687 raft_consensus.cc:2229] T cbc9e365873143cc955d8160a24a19b8 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:04.752542 5687 raft_consensus.cc:2258] T cbc9e365873143cc955d8160a24a19b8 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:04.753499 5687 ts_tablet_manager.cc:1875] T cbc9e365873143cc955d8160a24a19b8 P 9dec645b473f40578e726f3e50e219bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:04.755380 5696 tablet_replica.cc:330] T cbc9e365873143cc955d8160a24a19b8 P d1406e92aec84ac585304e57bd37a40d: stopping tablet replica
I20240502 21:27:04.755663 5696 raft_consensus.cc:2229] T cbc9e365873143cc955d8160a24a19b8 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:04.755939 5696 raft_consensus.cc:2258] T cbc9e365873143cc955d8160a24a19b8 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:04.756678 5697 tablet_replica.cc:330] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db: stopping tablet replica
I20240502 21:27:04.756992 5697 raft_consensus.cc:2229] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:04.757364 5697 raft_consensus.cc:2258] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:04.759747 5687 ts_tablet_manager.cc:1888] T cbc9e365873143cc955d8160a24a19b8 P 9dec645b473f40578e726f3e50e219bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:04.759904 5687 log.cc:1196] T cbc9e365873143cc955d8160a24a19b8 P 9dec645b473f40578e726f3e50e219bf: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal/wals/cbc9e365873143cc955d8160a24a19b8
I20240502 21:27:04.760314 5687 ts_tablet_manager.cc:1909] T cbc9e365873143cc955d8160a24a19b8 P 9dec645b473f40578e726f3e50e219bf: Deleting consensus metadata
I20240502 21:27:04.761687 5696 ts_tablet_manager.cc:1875] T cbc9e365873143cc955d8160a24a19b8 P d1406e92aec84ac585304e57bd37a40d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:04.762813 31379 catalog_manager.cc:4830] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): tablet cbc9e365873143cc955d8160a24a19b8 (table rebalancer_dynamic_table_new_name [id=dce95cf294b1486eaab9fd8fdf20de6d]) successfully deleted
I20240502 21:27:04.763433 5688 ts_tablet_manager.cc:1875] T 1f743c7345184feaa50871dc49a1f136 P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:04.763736 31796 consensus_queue.cc:237] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } }
I20240502 21:27:04.767298 1965 raft_consensus.cc:1261] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 9dec645b473f40578e726f3e50e219bf [term 1 LEARNER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:04.768072 5528 consensus_queue.cc:1035] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:04.777752 5696 ts_tablet_manager.cc:1888] T cbc9e365873143cc955d8160a24a19b8 P d1406e92aec84ac585304e57bd37a40d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:04.778122 31925 raft_consensus.cc:1261] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:04.779793 5528 consensus_queue.cc:1035] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:04.780226 5696 log.cc:1196] T cbc9e365873143cc955d8160a24a19b8 P d1406e92aec84ac585304e57bd37a40d: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal/wals/cbc9e365873143cc955d8160a24a19b8
I20240502 21:27:04.780673 5696 ts_tablet_manager.cc:1909] T cbc9e365873143cc955d8160a24a19b8 P d1406e92aec84ac585304e57bd37a40d: Deleting consensus metadata
I20240502 21:27:04.783166 31381 catalog_manager.cc:4830] TS d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): tablet cbc9e365873143cc955d8160a24a19b8 (table rebalancer_dynamic_table_new_name [id=dce95cf294b1486eaab9fd8fdf20de6d]) successfully deleted
I20240502 21:27:04.783224 5697 ts_tablet_manager.cc:1875] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:04.783773 5688 ts_tablet_manager.cc:1888] T 1f743c7345184feaa50871dc49a1f136 P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:27:04.783936 5688 log.cc:1196] T 1f743c7345184feaa50871dc49a1f136 P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/1f743c7345184feaa50871dc49a1f136
I20240502 21:27:04.784839 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet 1f743c7345184feaa50871dc49a1f136 (table rebalancer_extra_table_22 [id=27be64652cc14b9fa97100370e65428b]) successfully deleted
I20240502 21:27:04.787622 2094 raft_consensus.cc:1261] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:04.788360 5528 consensus_queue.cc:1035] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:04.789014 31923 raft_consensus.cc:2932] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:27:04.789939 31667 raft_consensus.cc:1261] T 62e70b8e69c94d76b23c655a3f5de5bd P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:04.791307 5677 consensus_queue.cc:1035] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:04.794078 5530 raft_consensus.cc:2932] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:27:04.797308 31388 catalog_manager.cc:5484] T c8b447d3c1a24051a914b7ed642808fa P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135). New cstate: current_term: 1 leader_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } }
W20240502 21:27:04.799039 31732 consensus_peers.cc:485] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c -> Peer 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): Couldn't send request to peer 1ade0354c44c4c37ae480c7a94b11e1b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62e70b8e69c94d76b23c655a3f5de5bd. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:04.799377 31394 catalog_manager.cc:5484] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: config changed from index 5 to 6, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:27:04.802523 31923 raft_consensus.cc:1261] T dd0198178da640688230aeb5a1106cb4 P 2d50f33557944bb480ea32199d9ee4f1 [term 2 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:27:04.803018 5678 consensus_queue.cc:1035] T dd0198178da640688230aeb5a1106cb4 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:04.806525 1965 raft_consensus.cc:2932] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:27:04.809007 31536 consensus_queue.cc:237] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:27:04.811193 31924 raft_consensus.cc:1261] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:04.811993 5528 consensus_queue.cc:1035] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:04.813439 1962 raft_consensus.cc:1261] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:04.814167 5528 consensus_queue.cc:1035] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:04.819510 2094 raft_consensus.cc:1261] T dd0198178da640688230aeb5a1106cb4 P 1ade0354c44c4c37ae480c7a94b11e1b [term 2 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:27:04.820128 5678 consensus_queue.cc:1035] T dd0198178da640688230aeb5a1106cb4 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:04.823361 5531 raft_consensus.cc:2932] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:27:04.828405 31382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5bc6c7211d5f4f73b2a050b9ec78c6b1 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:04.828927 31924 raft_consensus.cc:2932] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:27:04.830369 31533 raft_consensus.cc:1261] T 62e70b8e69c94d76b23c655a3f5de5bd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:04.830868 5678 consensus_queue.cc:1035] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:04.834633 1962 raft_consensus.cc:2932] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:27:04.836176 2092 raft_consensus.cc:2932] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9dec645b473f40578e726f3e50e219bf (127.24.226.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } }
I20240502 21:27:04.852008 31394 catalog_manager.cc:5484] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: config changed from index 6 to 7, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 21:27:04.853547 5698 log.cc:924] Time spent T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20240502 21:27:04.854028 31392 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:04.856303 4859 raft_consensus.cc:2932] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } } }
I20240502 21:27:04.856869 31665 raft_consensus.cc:2932] T 62e70b8e69c94d76b23c655a3f5de5bd P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } } }
I20240502 21:27:04.864814 5697 ts_tablet_manager.cc:1888] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:04.865044 5697 log.cc:1196] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal/wals/cbc9e365873143cc955d8160a24a19b8
I20240502 21:27:04.868672 2077 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bc6c7211d5f4f73b2a050b9ec78c6b1 with delete_type TABLET_DATA_TOMBSTONED (TS 1ade0354c44c4c37ae480c7a94b11e1b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:36740
I20240502 21:27:04.871071 5697 ts_tablet_manager.cc:1909] T cbc9e365873143cc955d8160a24a19b8 P 1e04157b98dd4fad98858c290b73c3db: Deleting consensus metadata
I20240502 21:27:04.872337 31381 catalog_manager.cc:4830] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): tablet cbc9e365873143cc955d8160a24a19b8 (table rebalancer_dynamic_table_new_name [id=dce95cf294b1486eaab9fd8fdf20de6d]) successfully deleted
I20240502 21:27:04.879199 31535 raft_consensus.cc:2932] T 62e70b8e69c94d76b23c655a3f5de5bd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } } }
I20240502 21:27:04.887516 31386 catalog_manager.cc:5484] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index -1 to 4, NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) added. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:04.897500 31775 tablet_service.cc:1467] Processing CreateTablet for tablet a6e2e40ab41f4faa80aa4485d4e8e4f6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f803fc85ce5940c9b595ddf0fe325413]), partition=
I20240502 21:27:04.897536 1814 tablet_service.cc:1467] Processing CreateTablet for tablet a6e2e40ab41f4faa80aa4485d4e8e4f6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f803fc85ce5940c9b595ddf0fe325413]), partition=
I20240502 21:27:04.897818 31775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6e2e40ab41f4faa80aa4485d4e8e4f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.897986 1814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6e2e40ab41f4faa80aa4485d4e8e4f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.904752 5647 tablet_bootstrap.cc:492] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:27:04.906170 5647 tablet_bootstrap.cc:654] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:04.908095 5647 tablet_bootstrap.cc:492] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:27:04.908227 5647 ts_tablet_manager.cc:1367] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240502 21:27:04.909544 5647 raft_consensus.cc:348] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:04.910081 5647 raft_consensus.cc:374] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:04.910271 5647 raft_consensus.cc:729] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:27:04.910620 5647 consensus_queue.cc:260] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:04.911221 5647 ts_tablet_manager.cc:1398] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:04.926309 5470 tablet_bootstrap.cc:492] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:27:04.927800 5470 tablet_bootstrap.cc:654] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:04.929240 31516 tablet_service.cc:1467] Processing CreateTablet for tablet a6e2e40ab41f4faa80aa4485d4e8e4f6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f803fc85ce5940c9b595ddf0fe325413]), partition=
I20240502 21:27:04.929750 31516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6e2e40ab41f4faa80aa4485d4e8e4f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.937012 5712 tablet_replica.cc:330] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1ade0354c44c4c37ae480c7a94b11e1b: stopping tablet replica
I20240502 21:27:04.937484 5069 raft_consensus.cc:1050] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada: attempting to promote NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b to VOTER
I20240502 21:27:04.938237 5069 consensus_queue.cc:237] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } }
I20240502 21:27:04.938745 5712 raft_consensus.cc:2229] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:04.938961 5712 raft_consensus.cc:2258] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:04.939579 5650 tablet_bootstrap.cc:492] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:27:04.940798 5650 tablet_bootstrap.cc:654] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:04.942359 5712 ts_tablet_manager.cc:1875] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:04.942441 5470 tablet_bootstrap.cc:492] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:27:04.942569 5470 ts_tablet_manager.cc:1367] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240502 21:27:04.943300 5470 raft_consensus.cc:348] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:04.943550 5470 raft_consensus.cc:374] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:04.943629 5470 raft_consensus.cc:729] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:27:04.943882 5470 consensus_queue.cc:260] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:04.945904 5530 raft_consensus.cc:1050] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f: attempting to promote NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada to VOTER
I20240502 21:27:04.945796 31796 consensus_queue.cc:237] T dd0198178da640688230aeb5a1106cb4 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } }
I20240502 21:27:04.948190 31925 raft_consensus.cc:1261] T dd0198178da640688230aeb5a1106cb4 P 2d50f33557944bb480ea32199d9ee4f1 [term 2 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 21:27:04.948743 4859 consensus_queue.cc:1035] T dd0198178da640688230aeb5a1106cb4 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:04.949916 31796 raft_consensus.cc:1261] T f61e34e65429408e83a13c7bc743e78a P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:04.950018 2092 raft_consensus.cc:1261] T f61e34e65429408e83a13c7bc743e78a P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:04.954260 5677 raft_consensus.cc:2932] T dd0198178da640688230aeb5a1106cb4 P 08d7a0680d244b55b93d17f9ab57b84c [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:27:04.954980 31924 raft_consensus.cc:2932] T dd0198178da640688230aeb5a1106cb4 P 2d50f33557944bb480ea32199d9ee4f1 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:27:04.955682 2094 raft_consensus.cc:1261] T dd0198178da640688230aeb5a1106cb4 P 1ade0354c44c4c37ae480c7a94b11e1b [term 2 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 21:27:04.956303 4559 consensus_queue.cc:1035] T dd0198178da640688230aeb5a1106cb4 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:04.957839 2092 raft_consensus.cc:2932] T dd0198178da640688230aeb5a1106cb4 P 1ade0354c44c4c37ae480c7a94b11e1b [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:27:04.946606 5530 consensus_queue.cc:237] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } }
I20240502 21:27:04.962910 31925 raft_consensus.cc:1261] T f61e34e65429408e83a13c7bc743e78a P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:04.963974 31394 catalog_manager.cc:5484] T dd0198178da640688230aeb5a1106cb4 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: config changed from index 5 to 7, VOTER a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) evicted. New cstate: current_term: 2 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:27:04.964835 1965 raft_consensus.cc:1261] T 4bc968ae4dcc4636a49257b6d5824bee P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:04.965435 5705 consensus_queue.cc:1035] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:04.966708 31381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dd0198178da640688230aeb5a1106cb4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:04.967468 31665 raft_consensus.cc:1261] T 4bc968ae4dcc4636a49257b6d5824bee P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:04.968144 5470 ts_tablet_manager.cc:1398] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20240502 21:27:04.968315 31516 tablet_service.cc:1514] Processing DeleteTablet for tablet dd0198178da640688230aeb5a1106cb4 with delete_type TABLET_DATA_TOMBSTONED (TS a44f41b0b6644fb6aa29476aed8c964f not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:45194
I20240502 21:27:04.968585 5705 consensus_queue.cc:1035] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:04.967257 2094 raft_consensus.cc:1261] T 4bc968ae4dcc4636a49257b6d5824bee P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:04.971889 5705 consensus_queue.cc:1035] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:04.982452 5201 consensus_queue.cc:1035] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:04.997072 5201 consensus_queue.cc:1035] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:04.998462 5201 consensus_queue.cc:1035] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:05.027817 5650 tablet_bootstrap.cc:492] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:27:05.028064 5650 ts_tablet_manager.cc:1367] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.089s user 0.008s sys 0.004s
I20240502 21:27:05.028813 5650 raft_consensus.cc:348] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:05.029127 5650 raft_consensus.cc:374] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:05.029220 5650 raft_consensus.cc:729] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.029515 5650 consensus_queue.cc:260] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:05.030372 5650 ts_tablet_manager.cc:1398] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:05.038311 2094 raft_consensus.cc:1261] T 9bc57b8a0388416290997df544e5cc0b P 1ade0354c44c4c37ae480c7a94b11e1b [term 2 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:27:05.039320 1833 raft_consensus.cc:1261] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [term 2 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:27:05.039808 1962 raft_consensus.cc:1261] T 9bc57b8a0388416290997df544e5cc0b P 9dec645b473f40578e726f3e50e219bf [term 2 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:27:05.040817 5417 consensus_queue.cc:1035] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:05.041347 4859 consensus_queue.cc:1035] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:05.041726 5417 consensus_queue.cc:1035] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:05.051405 5688 tablet_replica.cc:330] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:27:05.051642 5688 raft_consensus.cc:2229] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:05.053424 5688 raft_consensus.cc:2258] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:05.056355 5688 ts_tablet_manager.cc:1875] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:05.061151 5712 ts_tablet_manager.cc:1888] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1ade0354c44c4c37ae480c7a94b11e1b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 21:27:05.062764 5712 log.cc:1196] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal/wals/5bc6c7211d5f4f73b2a050b9ec78c6b1
I20240502 21:27:05.064114 5688 ts_tablet_manager.cc:1888] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 21:27:05.064082 31381 catalog_manager.cc:4830] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): tablet 5bc6c7211d5f4f73b2a050b9ec78c6b1 (table rebalancer_extra_table_6 [id=86898baa76944823b3c5e8aba98d8772]) successfully deleted
I20240502 21:27:05.064342 5688 log.cc:1196] T dd0198178da640688230aeb5a1106cb4 P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/dd0198178da640688230aeb5a1106cb4
I20240502 21:27:05.065223 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet dd0198178da640688230aeb5a1106cb4 (table rebalance_test_table_17 [id=2e8e8260e3b64486b5712ad7289d1cc8]) successfully deleted
I20240502 21:27:05.070549 31796 raft_consensus.cc:2932] T f61e34e65429408e83a13c7bc743e78a P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:27:05.079538 31394 catalog_manager.cc:5484] T f61e34e65429408e83a13c7bc743e78a P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:27:05.090837 5201 raft_consensus.cc:2932] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:27:05.101744 2226 consensus_queue.cc:237] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } }
I20240502 21:27:05.105553 31925 raft_consensus.cc:1261] T 883c729aa26f47178f40a17ad3b7e590 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:05.106153 1834 raft_consensus.cc:1261] T 883c729aa26f47178f40a17ad3b7e590 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:05.107028 5661 consensus_queue.cc:1035] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:05.113322 5522 consensus_queue.cc:1035] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:05.114121 5522 raft_consensus.cc:2932] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:27:05.115499 5512 consensus_queue.cc:1035] T 7cc4bbc0deb8464db065e23cda176437 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:05.117501 31925 raft_consensus.cc:2932] T 883c729aa26f47178f40a17ad3b7e590 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:27:05.127162 1834 raft_consensus.cc:2932] T 883c729aa26f47178f40a17ad3b7e590 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:27:05.138129 31381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 883c729aa26f47178f40a17ad3b7e590 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:05.140759 31388 catalog_manager.cc:5484] T 883c729aa26f47178f40a17ad3b7e590 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: config changed from index 3 to 4, VOTER 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) evicted. New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } } }
I20240502 21:27:05.150009 5706 raft_consensus.cc:2932] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:27:05.154182 1964 raft_consensus.cc:2932] T 4bc968ae4dcc4636a49257b6d5824bee P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:27:05.158422 31386 catalog_manager.cc:5484] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:05.165273 4904 consensus_queue.cc:1035] T c8b447d3c1a24051a914b7ed642808fa P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:05.167421 5678 raft_consensus.cc:482] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:05.167588 5678 raft_consensus.cc:504] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:05.167608 2094 raft_consensus.cc:2932] T 4bc968ae4dcc4636a49257b6d5824bee P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:27:05.168365 5678 leader_election.cc:288] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
W20240502 21:27:05.169384 31602 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.226.131:40807, user_credentials={real_user=slave}} blocked reactor thread for 91224.3us
I20240502 21:27:05.169950 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6e2e40ab41f4faa80aa4485d4e8e4f6" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:27:05.170168 31535 raft_consensus.cc:2454] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:27:05.170822 31733 leader_election.cc:302] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:27:05.171442 1831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6e2e40ab41f4faa80aa4485d4e8e4f6" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:27:05.171530 4859 raft_consensus.cc:2781] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:05.171635 4859 raft_consensus.cc:482] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:05.171630 1831 raft_consensus.cc:2454] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:27:05.171730 4859 raft_consensus.cc:3037] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:05.174681 4859 raft_consensus.cc:504] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:05.177610 31667 raft_consensus.cc:2932] T 4bc968ae4dcc4636a49257b6d5824bee P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:27:05.182777 1834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6e2e40ab41f4faa80aa4485d4e8e4f6" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:27:05.183034 1834 raft_consensus.cc:3037] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:05.184381 2096 raft_consensus.cc:2932] T f61e34e65429408e83a13c7bc743e78a P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:27:05.185966 1834 raft_consensus.cc:2454] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:27:05.186686 31732 leader_election.cc:302] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 1e04157b98dd4fad98858c290b73c3db; no voters:
I20240502 21:27:05.187728 5512 consensus_queue.cc:1035] T 7cc4bbc0deb8464db065e23cda176437 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:05.187726 31535 consensus_queue.cc:237] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } }
I20240502 21:27:05.190407 4559 raft_consensus.cc:2781] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:05.190419 2096 raft_consensus.cc:1261] T 4bc968ae4dcc4636a49257b6d5824bee P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:05.190539 4559 raft_consensus.cc:686] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Becoming Leader. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Running, Role: LEADER
I20240502 21:27:05.190771 4559 consensus_queue.cc:237] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:05.191470 5531 consensus_queue.cc:1035] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:05.203967 4859 leader_election.cc:288] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Requested vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:27:05.213862 5706 raft_consensus.cc:2932] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:27:05.214419 2096 raft_consensus.cc:2932] T 4bc968ae4dcc4636a49257b6d5824bee P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:27:05.219043 31382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4bc968ae4dcc4636a49257b6d5824bee with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:05.218910 31924 raft_consensus.cc:2932] T f61e34e65429408e83a13c7bc743e78a P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:27:05.220923 31533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6e2e40ab41f4faa80aa4485d4e8e4f6" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:27:05.221150 31533 raft_consensus.cc:3037] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:05.224267 31533 raft_consensus.cc:2454] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:27:05.229018 31389 catalog_manager.cc:5484] T 4bc968ae4dcc4636a49257b6d5824bee P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: config changed from index 3 to 4, VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) evicted. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:27:05.233083 1945 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bc968ae4dcc4636a49257b6d5824bee with delete_type TABLET_DATA_TOMBSTONED (TS 9dec645b473f40578e726f3e50e219bf not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33002
I20240502 21:27:05.233394 5687 tablet_replica.cc:330] T 4bc968ae4dcc4636a49257b6d5824bee P 9dec645b473f40578e726f3e50e219bf: stopping tablet replica
I20240502 21:27:05.233590 5687 raft_consensus.cc:2229] T 4bc968ae4dcc4636a49257b6d5824bee P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:05.233772 5687 raft_consensus.cc:2258] T 4bc968ae4dcc4636a49257b6d5824bee P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:05.235601 2077 tablet_service.cc:1514] Processing DeleteTablet for tablet 883c729aa26f47178f40a17ad3b7e590 with delete_type TABLET_DATA_TOMBSTONED (TS 1ade0354c44c4c37ae480c7a94b11e1b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36740
I20240502 21:27:05.235832 5687 ts_tablet_manager.cc:1875] T 4bc968ae4dcc4636a49257b6d5824bee P 9dec645b473f40578e726f3e50e219bf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:05.236205 5712 tablet_replica.cc:330] T 883c729aa26f47178f40a17ad3b7e590 P 1ade0354c44c4c37ae480c7a94b11e1b: stopping tablet replica
I20240502 21:27:05.236402 5712 raft_consensus.cc:2229] T 883c729aa26f47178f40a17ad3b7e590 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:05.236701 5712 raft_consensus.cc:2258] T 883c729aa26f47178f40a17ad3b7e590 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:05.238011 5712 ts_tablet_manager.cc:1875] T 883c729aa26f47178f40a17ad3b7e590 P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:05.244369 5687 ts_tablet_manager.cc:1888] T 4bc968ae4dcc4636a49257b6d5824bee P 9dec645b473f40578e726f3e50e219bf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:27:05.244307 4904 consensus_queue.cc:1035] T c8b447d3c1a24051a914b7ed642808fa P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:05.244571 5687 log.cc:1196] T 4bc968ae4dcc4636a49257b6d5824bee P 9dec645b473f40578e726f3e50e219bf: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal/wals/4bc968ae4dcc4636a49257b6d5824bee
I20240502 21:27:05.246003 31379 catalog_manager.cc:4830] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): tablet 4bc968ae4dcc4636a49257b6d5824bee (table rebalancer_extra_table_22 [id=27be64652cc14b9fa97100370e65428b]) successfully deleted
I20240502 21:27:05.246503 5712 ts_tablet_manager.cc:1888] T 883c729aa26f47178f40a17ad3b7e590 P 1ade0354c44c4c37ae480c7a94b11e1b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:27:05.246711 5712 log.cc:1196] T 883c729aa26f47178f40a17ad3b7e590 P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal/wals/883c729aa26f47178f40a17ad3b7e590
I20240502 21:27:05.247701 31381 catalog_manager.cc:4830] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): tablet 883c729aa26f47178f40a17ad3b7e590 (table rebalancer_extra_table_16 [id=cc412415870b4447b2e9707c2f7fd323]) successfully deleted
I20240502 21:27:05.254648 31535 consensus_queue.cc:237] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } }
I20240502 21:27:05.259136 31665 consensus_queue.cc:237] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } }
I20240502 21:27:05.265852 5741 ts_tablet_manager.cc:902] T 62e70b8e69c94d76b23c655a3f5de5bd P 1ade0354c44c4c37ae480c7a94b11e1b: Initiating tablet copy from peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:27:05.269685 31924 raft_consensus.cc:1261] T 988a0a9425434cd0b724710e9ebf23a6 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:05.271479 31796 raft_consensus.cc:1261] T 988a0a9425434cd0b724710e9ebf23a6 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:05.271991 5531 consensus_queue.cc:1035] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:05.272274 5705 consensus_queue.cc:1035] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:27:05.273916 31471 consensus_peers.cc:485] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f -> Peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): Couldn't send request to peer 1e04157b98dd4fad98858c290b73c3db. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 988a0a9425434cd0b724710e9ebf23a6. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:05.286773 31797 raft_consensus.cc:1261] T f61e34e65429408e83a13c7bc743e78a P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:05.285446 2096 raft_consensus.cc:1261] T f61e34e65429408e83a13c7bc743e78a P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:05.292691 31388 catalog_manager.cc:5484] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: term changed from 0 to 1, leader changed from <none> to 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131). New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:05.294163 5726 consensus_queue.cc:1035] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:05.298282 31667 raft_consensus.cc:1261] T 4bc968ae4dcc4636a49257b6d5824bee P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:05.300230 5531 consensus_queue.cc:1035] T 4bc968ae4dcc4636a49257b6d5824bee P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:05.302120 5726 consensus_queue.cc:1035] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:05.302996 5721 raft_consensus.cc:2932] T f61e34e65429408e83a13c7bc743e78a P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:27:05.308774 31382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f61e34e65429408e83a13c7bc743e78a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:05.308454 5531 raft_consensus.cc:2932] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } } }
I20240502 21:27:05.309325 31667 raft_consensus.cc:2932] T 4bc968ae4dcc4636a49257b6d5824bee P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:27:05.315290 31796 raft_consensus.cc:2932] T 988a0a9425434cd0b724710e9ebf23a6 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } } }
I20240502 21:27:05.316706 31793 raft_consensus.cc:2932] T f61e34e65429408e83a13c7bc743e78a P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:27:05.323983 31797 consensus_queue.cc:237] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } }
I20240502 21:27:05.326061 2093 raft_consensus.cc:2932] T f61e34e65429408e83a13c7bc743e78a P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:27:05.327131 1962 raft_consensus.cc:1261] T 9bc57b8a0388416290997df544e5cc0b P 9dec645b473f40578e726f3e50e219bf [term 2 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 21:27:05.327764 4559 consensus_queue.cc:1035] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:05.328055 2096 raft_consensus.cc:1261] T 9bc57b8a0388416290997df544e5cc0b P 1ade0354c44c4c37ae480c7a94b11e1b [term 2 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 21:27:05.328639 4859 consensus_queue.cc:1035] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:05.329697 31392 catalog_manager.cc:5484] T 988a0a9425434cd0b724710e9ebf23a6 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index -1 to 4, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } } }
I20240502 21:27:05.326711 31924 raft_consensus.cc:2932] T 988a0a9425434cd0b724710e9ebf23a6 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } } }
I20240502 21:27:05.330241 31392 catalog_manager.cc:5484] T f61e34e65429408e83a13c7bc743e78a P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index 5 to 6, VOTER 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132) evicted. New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:27:05.331156 4859 raft_consensus.cc:2932] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:27:05.334874 2096 raft_consensus.cc:2932] T 9bc57b8a0388416290997df544e5cc0b P 1ade0354c44c4c37ae480c7a94b11e1b [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:27:05.335399 31906 tablet_service.cc:1514] Processing DeleteTablet for tablet f61e34e65429408e83a13c7bc743e78a with delete_type TABLET_DATA_TOMBSTONED (TS 2d50f33557944bb480ea32199d9ee4f1 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47824
I20240502 21:27:05.336748 5653 consensus_queue.cc:1035] T e84560e2ba25443cbba6babbbfcaf104 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:05.341697 31381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9bc57b8a0388416290997df544e5cc0b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:05.345659 5643 consensus_queue.cc:1035] T e84560e2ba25443cbba6babbbfcaf104 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:05.350486 31393 catalog_manager.cc:5484] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: config changed from index 3 to 5, VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) evicted. New cstate: current_term: 2 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:05.355671 1814 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bc57b8a0388416290997df544e5cc0b with delete_type TABLET_DATA_TOMBSTONED (TS 1e04157b98dd4fad98858c290b73c3db not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47510
I20240502 21:27:05.355975 5697 tablet_replica.cc:330] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db: stopping tablet replica
I20240502 21:27:05.356211 5697 raft_consensus.cc:2229] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:05.356362 5697 raft_consensus.cc:2258] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:05.357446 31388 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_extra_table_44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:05.359360 5697 ts_tablet_manager.cc:1875] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:05.362244 5754 tablet_replica.cc:330] T f61e34e65429408e83a13c7bc743e78a P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:27:05.362411 5754 raft_consensus.cc:2229] T f61e34e65429408e83a13c7bc743e78a P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:05.362665 5754 raft_consensus.cc:2258] T f61e34e65429408e83a13c7bc743e78a P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:05.364660 5754 ts_tablet_manager.cc:1875] T f61e34e65429408e83a13c7bc743e78a P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:05.374697 1964 raft_consensus.cc:2932] T 9bc57b8a0388416290997df544e5cc0b P 9dec645b473f40578e726f3e50e219bf [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } } }
I20240502 21:27:05.376914 5754 ts_tablet_manager.cc:1888] T f61e34e65429408e83a13c7bc743e78a P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:05.376996 5754 log.cc:1196] T f61e34e65429408e83a13c7bc743e78a P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/f61e34e65429408e83a13c7bc743e78a
I20240502 21:27:05.377709 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet f61e34e65429408e83a13c7bc743e78a (table rebalance_test_table_19 [id=654ed3d4f1ef4492ac274c10f2ba1f3d]) successfully deleted
I20240502 21:27:05.388588 31645 tablet_service.cc:1467] Processing CreateTablet for tablet 78f06bc68b954dc4b55b3c8c3c612c2f (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=ac2a372213e84d6a961653074f499ead]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:05.389007 31645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78f06bc68b954dc4b55b3c8c3c612c2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.389252 2077 tablet_service.cc:1467] Processing CreateTablet for tablet 04070a3231524eac83e0c94bf2aee585 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=ac2a372213e84d6a961653074f499ead]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:05.389652 2077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04070a3231524eac83e0c94bf2aee585. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.392772 31516 tablet_service.cc:1467] Processing CreateTablet for tablet 78f06bc68b954dc4b55b3c8c3c612c2f (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=ac2a372213e84d6a961653074f499ead]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:05.393118 31516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78f06bc68b954dc4b55b3c8c3c612c2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.393118 31775 tablet_service.cc:1467] Processing CreateTablet for tablet 4d5d4121e8284195b76bfd12f826f942 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=ac2a372213e84d6a961653074f499ead]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:05.393110 1946 tablet_service.cc:1467] Processing CreateTablet for tablet 4d5d4121e8284195b76bfd12f826f942 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=ac2a372213e84d6a961653074f499ead]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:05.393494 31775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d5d4121e8284195b76bfd12f826f942. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.393517 1946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d5d4121e8284195b76bfd12f826f942. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.396049 5697 ts_tablet_manager.cc:1888] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 21:27:05.396188 5697 log.cc:1196] T 9bc57b8a0388416290997df544e5cc0b P 1e04157b98dd4fad98858c290b73c3db: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal/wals/9bc57b8a0388416290997df544e5cc0b
I20240502 21:27:05.396739 1945 tablet_service.cc:1467] Processing CreateTablet for tablet 78f06bc68b954dc4b55b3c8c3c612c2f (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=ac2a372213e84d6a961653074f499ead]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:05.397116 1945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78f06bc68b954dc4b55b3c8c3c612c2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.398854 1814 tablet_service.cc:1467] Processing CreateTablet for tablet 04070a3231524eac83e0c94bf2aee585 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=ac2a372213e84d6a961653074f499ead]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:05.399243 1814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04070a3231524eac83e0c94bf2aee585. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.400808 31514 tablet_service.cc:1467] Processing CreateTablet for tablet 4d5d4121e8284195b76bfd12f826f942 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=ac2a372213e84d6a961653074f499ead]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:05.401247 31514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d5d4121e8284195b76bfd12f826f942. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.402055 31381 catalog_manager.cc:4830] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): tablet 9bc57b8a0388416290997df544e5cc0b (table rebalancer_extra_table_6 [id=86898baa76944823b3c5e8aba98d8772]) successfully deleted
I20240502 21:27:05.405371 2208 tablet_service.cc:1467] Processing CreateTablet for tablet 04070a3231524eac83e0c94bf2aee585 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=ac2a372213e84d6a961653074f499ead]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:05.405874 2208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04070a3231524eac83e0c94bf2aee585. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.413267 5759 tablet_bootstrap.cc:492] T 4d5d4121e8284195b76bfd12f826f942 P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:27:05.416471 5759 tablet_bootstrap.cc:654] T 4d5d4121e8284195b76bfd12f826f942 P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:05.417203 5470 tablet_bootstrap.cc:492] T 04070a3231524eac83e0c94bf2aee585 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:27:05.418749 5470 tablet_bootstrap.cc:654] T 04070a3231524eac83e0c94bf2aee585 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:05.419638 5650 tablet_bootstrap.cc:492] T 78f06bc68b954dc4b55b3c8c3c612c2f P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:27:05.421401 5650 tablet_bootstrap.cc:654] T 78f06bc68b954dc4b55b3c8c3c612c2f P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:05.431084 5759 tablet_bootstrap.cc:492] T 4d5d4121e8284195b76bfd12f826f942 P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:27:05.431313 5759 ts_tablet_manager.cc:1367] T 4d5d4121e8284195b76bfd12f826f942 P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.004s
I20240502 21:27:05.432194 5759 raft_consensus.cc:348] T 4d5d4121e8284195b76bfd12f826f942 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:05.432549 5759 raft_consensus.cc:374] T 4d5d4121e8284195b76bfd12f826f942 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:05.432691 5759 raft_consensus.cc:729] T 4d5d4121e8284195b76bfd12f826f942 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.432971 5759 consensus_queue.cc:260] T 4d5d4121e8284195b76bfd12f826f942 P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:05.435230 5759 ts_tablet_manager.cc:1398] T 4d5d4121e8284195b76bfd12f826f942 P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 21:27:05.435560 5759 tablet_bootstrap.cc:492] T 78f06bc68b954dc4b55b3c8c3c612c2f P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
I20240502 21:27:05.437178 5759 tablet_bootstrap.cc:654] T 78f06bc68b954dc4b55b3c8c3c612c2f P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:05.448541 5759 tablet_bootstrap.cc:492] T 78f06bc68b954dc4b55b3c8c3c612c2f P 9dec645b473f40578e726f3e50e219bf: No bootstrap required, opened a new log
I20240502 21:27:05.448746 5759 ts_tablet_manager.cc:1367] T 78f06bc68b954dc4b55b3c8c3c612c2f P 9dec645b473f40578e726f3e50e219bf: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20240502 21:27:05.449620 5759 raft_consensus.cc:348] T 78f06bc68b954dc4b55b3c8c3c612c2f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:05.449895 5759 raft_consensus.cc:374] T 78f06bc68b954dc4b55b3c8c3c612c2f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:05.450083 5759 raft_consensus.cc:729] T 78f06bc68b954dc4b55b3c8c3c612c2f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.450371 5759 consensus_queue.cc:260] T 78f06bc68b954dc4b55b3c8c3c612c2f P 9dec645b473f40578e726f3e50e219bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:05.450583 5767 tablet_bootstrap.cc:492] T 78f06bc68b954dc4b55b3c8c3c612c2f P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:27:05.452580 5767 tablet_bootstrap.cc:654] T 78f06bc68b954dc4b55b3c8c3c612c2f P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:05.452608 5759 ts_tablet_manager.cc:1398] T 78f06bc68b954dc4b55b3c8c3c612c2f P 9dec645b473f40578e726f3e50e219bf: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:05.454398 5741 tablet_copy_client.cc:308] T 62e70b8e69c94d76b23c655a3f5de5bd P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.131:40807
I20240502 21:27:05.454466 5761 tablet_bootstrap.cc:492] T 04070a3231524eac83e0c94bf2aee585 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:27:05.455077 31817 tablet_copy_service.cc:140] P 08d7a0680d244b55b93d17f9ab57b84c: Received BeginTabletCopySession request for tablet 62e70b8e69c94d76b23c655a3f5de5bd from peer 1ade0354c44c4c37ae480c7a94b11e1b ({username='slave'} at 127.24.226.135:48019)
I20240502 21:27:05.455261 31817 tablet_copy_service.cc:161] P 08d7a0680d244b55b93d17f9ab57b84c: Beginning new tablet copy session on tablet 62e70b8e69c94d76b23c655a3f5de5bd from peer 1ade0354c44c4c37ae480c7a94b11e1b at {username='slave'} at 127.24.226.135:48019: session id = 1ade0354c44c4c37ae480c7a94b11e1b-62e70b8e69c94d76b23c655a3f5de5bd
I20240502 21:27:05.456132 5470 tablet_bootstrap.cc:492] T 04070a3231524eac83e0c94bf2aee585 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:27:05.456887 5650 tablet_bootstrap.cc:492] T 78f06bc68b954dc4b55b3c8c3c612c2f P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:27:05.457163 5650 ts_tablet_manager.cc:1367] T 78f06bc68b954dc4b55b3c8c3c612c2f P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.038s user 0.000s sys 0.012s
I20240502 21:27:05.457996 31817 tablet_copy_source_session.cc:205] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:05.458214 5650 raft_consensus.cc:348] T 78f06bc68b954dc4b55b3c8c3c612c2f P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:05.458474 5650 raft_consensus.cc:374] T 78f06bc68b954dc4b55b3c8c3c612c2f P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:05.458611 5650 raft_consensus.cc:729] T 78f06bc68b954dc4b55b3c8c3c612c2f P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.458767 31388 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:05.458882 5650 consensus_queue.cc:260] T 78f06bc68b954dc4b55b3c8c3c612c2f P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:05.459602 5650 ts_tablet_manager.cc:1398] T 78f06bc68b954dc4b55b3c8c3c612c2f P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:27:05.460530 5766 tablet_bootstrap.cc:492] T 4d5d4121e8284195b76bfd12f826f942 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:27:05.461016 5650 tablet_bootstrap.cc:492] T 4d5d4121e8284195b76bfd12f826f942 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:27:05.461282 5765 tablet_bootstrap.cc:492] T 04070a3231524eac83e0c94bf2aee585 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:27:05.462697 5650 tablet_bootstrap.cc:654] T 4d5d4121e8284195b76bfd12f826f942 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:05.466171 5767 tablet_bootstrap.cc:492] T 78f06bc68b954dc4b55b3c8c3c612c2f P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:27:05.466293 5767 ts_tablet_manager.cc:1367] T 78f06bc68b954dc4b55b3c8c3c612c2f P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.003s
I20240502 21:27:05.467000 5767 raft_consensus.cc:348] T 78f06bc68b954dc4b55b3c8c3c612c2f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:05.467193 5767 raft_consensus.cc:374] T 78f06bc68b954dc4b55b3c8c3c612c2f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:05.467247 5767 raft_consensus.cc:729] T 78f06bc68b954dc4b55b3c8c3c612c2f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.467487 5767 consensus_queue.cc:260] T 78f06bc68b954dc4b55b3c8c3c612c2f P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:05.468143 5767 ts_tablet_manager.cc:1398] T 78f06bc68b954dc4b55b3c8c3c612c2f P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:05.472159 31535 raft_consensus.cc:1261] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:05.473802 5761 tablet_bootstrap.cc:654] T 04070a3231524eac83e0c94bf2aee585 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:05.472707 1833 raft_consensus.cc:1261] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:05.479245 5470 ts_tablet_manager.cc:1367] T 04070a3231524eac83e0c94bf2aee585 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.009s
I20240502 21:27:05.479557 5741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62e70b8e69c94d76b23c655a3f5de5bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.480093 5761 tablet_bootstrap.cc:492] T 04070a3231524eac83e0c94bf2aee585 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:27:05.480283 5761 ts_tablet_manager.cc:1367] T 04070a3231524eac83e0c94bf2aee585 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20240502 21:27:05.481519 5761 raft_consensus.cc:348] T 04070a3231524eac83e0c94bf2aee585 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:05.481819 5761 raft_consensus.cc:374] T 04070a3231524eac83e0c94bf2aee585 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:05.481969 5761 raft_consensus.cc:729] T 04070a3231524eac83e0c94bf2aee585 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.482302 5761 consensus_queue.cc:260] T 04070a3231524eac83e0c94bf2aee585 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:05.473543 4859 consensus_queue.cc:1035] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:05.482854 5765 tablet_bootstrap.cc:654] T 04070a3231524eac83e0c94bf2aee585 P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:05.483194 5761 ts_tablet_manager.cc:1398] T 04070a3231524eac83e0c94bf2aee585 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:27:05.483258 4559 consensus_queue.cc:1035] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:05.498674 5650 tablet_bootstrap.cc:492] T 4d5d4121e8284195b76bfd12f826f942 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:27:05.501511 5650 ts_tablet_manager.cc:1367] T 4d5d4121e8284195b76bfd12f826f942 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.041s user 0.010s sys 0.016s
I20240502 21:27:05.502138 5650 raft_consensus.cc:348] T 4d5d4121e8284195b76bfd12f826f942 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:05.502491 5650 raft_consensus.cc:374] T 4d5d4121e8284195b76bfd12f826f942 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:05.502588 5650 raft_consensus.cc:729] T 4d5d4121e8284195b76bfd12f826f942 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.502810 5650 consensus_queue.cc:260] T 4d5d4121e8284195b76bfd12f826f942 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:05.503393 5650 ts_tablet_manager.cc:1398] T 4d5d4121e8284195b76bfd12f826f942 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:27:05.505162 5781 tablet.cc:1722] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:05.505776 5781 tablet.cc:1620] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 1e04157b98dd4fad98858c290b73c3db: MemRowSet was empty: no flush needed.
I20240502 21:27:05.506887 5782 tablet.cc:1722] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P a44f41b0b6644fb6aa29476aed8c964f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:05.507544 5782 tablet.cc:1620] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P a44f41b0b6644fb6aa29476aed8c964f: MemRowSet was empty: no flush needed.
I20240502 21:27:05.509016 31388 catalog_manager.cc:5576] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039) does not have the latest schema for tablet a6e2e40ab41f4faa80aa4485d4e8e4f6 (table rebalancer_dynamic_table [id=f803fc85ce5940c9b595ddf0fe325413]). Expected version 1 got 0
I20240502 21:27:05.516196 5765 tablet_bootstrap.cc:492] T 04070a3231524eac83e0c94bf2aee585 P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:27:05.516536 5765 ts_tablet_manager.cc:1367] T 04070a3231524eac83e0c94bf2aee585 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.059s user 0.013s sys 0.000s
I20240502 21:27:05.516901 31392 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=f803fc85ce5940c9b595ddf0fe325413] alter complete (version 1)
I20240502 21:27:05.518530 5765 raft_consensus.cc:348] T 04070a3231524eac83e0c94bf2aee585 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:05.519809 5765 raft_consensus.cc:374] T 04070a3231524eac83e0c94bf2aee585 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:05.519989 5765 raft_consensus.cc:729] T 04070a3231524eac83e0c94bf2aee585 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.520295 5765 consensus_queue.cc:260] T 04070a3231524eac83e0c94bf2aee585 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:05.520962 5765 ts_tablet_manager.cc:1398] T 04070a3231524eac83e0c94bf2aee585 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:05.529640 5470 raft_consensus.cc:348] T 04070a3231524eac83e0c94bf2aee585 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:05.541252 31388 catalog_manager.cc:5576] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863) does not have the latest schema for tablet a6e2e40ab41f4faa80aa4485d4e8e4f6 (table rebalancer_dynamic_table [id=f803fc85ce5940c9b595ddf0fe325413]). Expected version 1 got 0
I20240502 21:27:05.541682 5470 raft_consensus.cc:374] T 04070a3231524eac83e0c94bf2aee585 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:05.541860 5766 tablet_bootstrap.cc:654] T 4d5d4121e8284195b76bfd12f826f942 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:05.541858 5470 raft_consensus.cc:729] T 04070a3231524eac83e0c94bf2aee585 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.542089 5470 consensus_queue.cc:260] T 04070a3231524eac83e0c94bf2aee585 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:05.542809 5345 raft_consensus.cc:482] T 78f06bc68b954dc4b55b3c8c3c612c2f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:05.543006 5345 raft_consensus.cc:504] T 78f06bc68b954dc4b55b3c8c3c612c2f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:05.543814 5345 leader_election.cc:288] T 78f06bc68b954dc4b55b3c8c3c612c2f P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:27:05.542667 5470 ts_tablet_manager.cc:1398] T 04070a3231524eac83e0c94bf2aee585 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.063s user 0.001s sys 0.001s
I20240502 21:27:05.545219 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f06bc68b954dc4b55b3c8c3c612c2f" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:27:05.545536 31535 raft_consensus.cc:2454] T 78f06bc68b954dc4b55b3c8c3c612c2f P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 0.
I20240502 21:27:05.546054 5780 tablet.cc:1722] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:05.546445 31664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f06bc68b954dc4b55b3c8c3c612c2f" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:27:05.546646 31664 raft_consensus.cc:2454] T 78f06bc68b954dc4b55b3c8c3c612c2f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 0.
I20240502 21:27:05.548069 1902 leader_election.cc:302] T 78f06bc68b954dc4b55b3c8c3c612c2f P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:27:05.549650 5780 tablet.cc:1620] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c: MemRowSet was empty: no flush needed.
I20240502 21:27:05.549718 31388 catalog_manager.cc:5576] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807) does not have the latest schema for tablet a6e2e40ab41f4faa80aa4485d4e8e4f6 (table rebalancer_dynamic_table [id=f803fc85ce5940c9b595ddf0fe325413]). Expected version 1 got 0
I20240502 21:27:05.552512 5345 raft_consensus.cc:2781] T 78f06bc68b954dc4b55b3c8c3c612c2f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:05.552604 5345 raft_consensus.cc:482] T 78f06bc68b954dc4b55b3c8c3c612c2f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:05.552708 5345 raft_consensus.cc:3037] T 78f06bc68b954dc4b55b3c8c3c612c2f P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:05.553122 31666 consensus_queue.cc:237] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:27:05.553586 5741 tablet_copy_client.cc:783] T 62e70b8e69c94d76b23c655a3f5de5bd P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:05.553894 5741 tablet_copy_client.cc:647] T 62e70b8e69c94d76b23c655a3f5de5bd P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:05.555512 5345 raft_consensus.cc:504] T 78f06bc68b954dc4b55b3c8c3c612c2f P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:05.557340 5741 tablet_copy_client.cc:523] T 62e70b8e69c94d76b23c655a3f5de5bd P 1ade0354c44c4c37ae480c7a94b11e1b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:05.557593 31665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f06bc68b954dc4b55b3c8c3c612c2f" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:27:05.557744 31665 raft_consensus.cc:3037] T 78f06bc68b954dc4b55b3c8c3c612c2f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:05.561448 5741 tablet_bootstrap.cc:492] T 62e70b8e69c94d76b23c655a3f5de5bd P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
W20240502 21:27:05.563016 31600 consensus_peers.cc:485] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 043f33cb58084c3ebbce4893d1b63fbc. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:05.564061 31533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f06bc68b954dc4b55b3c8c3c612c2f" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:27:05.564208 31536 raft_consensus.cc:1261] T 043f33cb58084c3ebbce4893d1b63fbc P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:05.564293 31533 raft_consensus.cc:3037] T 78f06bc68b954dc4b55b3c8c3c612c2f P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:05.564517 31796 raft_consensus.cc:1261] T 043f33cb58084c3ebbce4893d1b63fbc P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:05.564745 5735 consensus_queue.cc:1035] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:05.565099 5721 consensus_queue.cc:1035] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:05.568230 31665 raft_consensus.cc:2454] T 78f06bc68b954dc4b55b3c8c3c612c2f P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 1.
I20240502 21:27:05.568715 1902 leader_election.cc:302] T 78f06bc68b954dc4b55b3c8c3c612c2f P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 9dec645b473f40578e726f3e50e219bf; no voters:
I20240502 21:27:05.568985 5537 raft_consensus.cc:2781] T 78f06bc68b954dc4b55b3c8c3c612c2f P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:05.570150 5345 leader_election.cc:288] T 78f06bc68b954dc4b55b3c8c3c612c2f P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 election: Requested vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:27:05.570401 31533 raft_consensus.cc:2454] T 78f06bc68b954dc4b55b3c8c3c612c2f P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 1.
I20240502 21:27:05.573760 5345 raft_consensus.cc:482] T 4d5d4121e8284195b76bfd12f826f942 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:05.573935 5345 raft_consensus.cc:504] T 4d5d4121e8284195b76bfd12f826f942 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:05.575073 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d5d4121e8284195b76bfd12f826f942" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:27:05.575663 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d5d4121e8284195b76bfd12f826f942" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:27:05.575933 31535 raft_consensus.cc:2454] T 4d5d4121e8284195b76bfd12f826f942 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 0.
I20240502 21:27:05.576198 5345 leader_election.cc:288] T 4d5d4121e8284195b76bfd12f826f942 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
W20240502 21:27:05.576323 1901 leader_election.cc:341] T 4d5d4121e8284195b76bfd12f826f942 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:05.578289 1902 leader_election.cc:302] T 4d5d4121e8284195b76bfd12f826f942 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9dec645b473f40578e726f3e50e219bf, a44f41b0b6644fb6aa29476aed8c964f; no voters: 08d7a0680d244b55b93d17f9ab57b84c
I20240502 21:27:05.579761 5345 raft_consensus.cc:2781] T 4d5d4121e8284195b76bfd12f826f942 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:05.579936 5345 raft_consensus.cc:482] T 4d5d4121e8284195b76bfd12f826f942 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:05.580116 5345 raft_consensus.cc:3037] T 4d5d4121e8284195b76bfd12f826f942 P 9dec645b473f40578e726f3e50e219bf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:05.581488 5721 raft_consensus.cc:2932] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:27:05.583235 5345 raft_consensus.cc:504] T 4d5d4121e8284195b76bfd12f826f942 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:05.584154 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d5d4121e8284195b76bfd12f826f942" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
W20240502 21:27:05.584694 1901 leader_election.cc:341] T 4d5d4121e8284195b76bfd12f826f942 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:05.585431 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d5d4121e8284195b76bfd12f826f942" candidate_uuid: "9dec645b473f40578e726f3e50e219bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:27:05.585613 31535 raft_consensus.cc:3037] T 4d5d4121e8284195b76bfd12f826f942 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:05.585328 31533 raft_consensus.cc:2932] T 043f33cb58084c3ebbce4893d1b63fbc P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:27:05.591276 31392 catalog_manager.cc:5484] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:05.597110 5537 raft_consensus.cc:686] T 78f06bc68b954dc4b55b3c8c3c612c2f P 9dec645b473f40578e726f3e50e219bf [term 1 LEADER]: Becoming Leader. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Running, Role: LEADER
I20240502 21:27:05.596863 31797 raft_consensus.cc:2932] T 043f33cb58084c3ebbce4893d1b63fbc P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:27:05.597524 5537 consensus_queue.cc:237] T 78f06bc68b954dc4b55b3c8c3c612c2f P 9dec645b473f40578e726f3e50e219bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:05.613533 31389 catalog_manager.cc:5484] T 78f06bc68b954dc4b55b3c8c3c612c2f P 9dec645b473f40578e726f3e50e219bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dec645b473f40578e726f3e50e219bf (127.24.226.134). New cstate: current_term: 1 leader_uuid: "9dec645b473f40578e726f3e50e219bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:05.621541 5345 leader_election.cc:288] T 4d5d4121e8284195b76bfd12f826f942 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 election: Requested vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:27:05.621767 5766 tablet_bootstrap.cc:492] T 4d5d4121e8284195b76bfd12f826f942 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:27:05.624204 5766 ts_tablet_manager.cc:1367] T 4d5d4121e8284195b76bfd12f826f942 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.163s user 0.007s sys 0.000s
I20240502 21:27:05.624851 31389 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:05.626438 5766 raft_consensus.cc:348] T 4d5d4121e8284195b76bfd12f826f942 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:05.627388 5766 raft_consensus.cc:374] T 4d5d4121e8284195b76bfd12f826f942 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:05.627530 5766 raft_consensus.cc:729] T 4d5d4121e8284195b76bfd12f826f942 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:27:05.627763 5766 consensus_queue.cc:260] T 4d5d4121e8284195b76bfd12f826f942 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:05.630964 5766 ts_tablet_manager.cc:1398] T 4d5d4121e8284195b76bfd12f826f942 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 21:27:05.635402 5780 tablet.cc:1722] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:05.635922 5780 tablet.cc:1620] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c: MemRowSet was empty: no flush needed.
I20240502 21:27:05.638440 5781 tablet.cc:1722] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:05.638881 5781 tablet.cc:1620] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 1e04157b98dd4fad98858c290b73c3db: MemRowSet was empty: no flush needed.
I20240502 21:27:05.639801 5741 tablet_bootstrap.cc:492] T 62e70b8e69c94d76b23c655a3f5de5bd P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:05.640223 5741 tablet_bootstrap.cc:492] T 62e70b8e69c94d76b23c655a3f5de5bd P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap complete.
I20240502 21:27:05.640750 5741 ts_tablet_manager.cc:1367] T 62e70b8e69c94d76b23c655a3f5de5bd P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.079s user 0.038s sys 0.010s
I20240502 21:27:05.641453 5741 raft_consensus.cc:348] T 62e70b8e69c94d76b23c655a3f5de5bd P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } }
I20240502 21:27:05.641712 5741 raft_consensus.cc:729] T 62e70b8e69c94d76b23c655a3f5de5bd P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: LEARNER
I20240502 21:27:05.641870 5741 consensus_queue.cc:260] T 62e70b8e69c94d76b23c655a3f5de5bd P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: NON_VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: true } }
I20240502 21:27:05.643906 5741 ts_tablet_manager.cc:1398] T 62e70b8e69c94d76b23c655a3f5de5bd P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 21:27:05.645545 31817 tablet_copy_service.cc:342] P 08d7a0680d244b55b93d17f9ab57b84c: Request end of tablet copy session 1ade0354c44c4c37ae480c7a94b11e1b-62e70b8e69c94d76b23c655a3f5de5bd received from {username='slave'} at 127.24.226.135:48019
I20240502 21:27:05.645679 31817 tablet_copy_service.cc:433] P 08d7a0680d244b55b93d17f9ab57b84c: ending tablet copy session 1ade0354c44c4c37ae480c7a94b11e1b-62e70b8e69c94d76b23c655a3f5de5bd on tablet 62e70b8e69c94d76b23c655a3f5de5bd with peer 1ade0354c44c4c37ae480c7a94b11e1b
I20240502 21:27:05.649784 5782 tablet.cc:1722] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P a44f41b0b6644fb6aa29476aed8c964f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:05.650252 5782 tablet.cc:1620] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P a44f41b0b6644fb6aa29476aed8c964f: MemRowSet was empty: no flush needed.
I20240502 21:27:05.651270 31392 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=f803fc85ce5940c9b595ddf0fe325413] alter complete (version 2)
I20240502 21:27:05.588485 31535 raft_consensus.cc:2454] T 4d5d4121e8284195b76bfd12f826f942 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dec645b473f40578e726f3e50e219bf in term 1.
I20240502 21:27:05.687412 1902 leader_election.cc:302] T 4d5d4121e8284195b76bfd12f826f942 P 9dec645b473f40578e726f3e50e219bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9dec645b473f40578e726f3e50e219bf, a44f41b0b6644fb6aa29476aed8c964f; no voters: 08d7a0680d244b55b93d17f9ab57b84c
I20240502 21:27:05.687834 5537 raft_consensus.cc:2781] T 4d5d4121e8284195b76bfd12f826f942 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:05.687971 5537 raft_consensus.cc:686] T 4d5d4121e8284195b76bfd12f826f942 P 9dec645b473f40578e726f3e50e219bf [term 1 LEADER]: Becoming Leader. State: Replica: 9dec645b473f40578e726f3e50e219bf, State: Running, Role: LEADER
I20240502 21:27:05.688208 5537 consensus_queue.cc:237] T 4d5d4121e8284195b76bfd12f826f942 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:05.700273 31389 catalog_manager.cc:5484] T 4d5d4121e8284195b76bfd12f826f942 P 9dec645b473f40578e726f3e50e219bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dec645b473f40578e726f3e50e219bf (127.24.226.134). New cstate: current_term: 1 leader_uuid: "9dec645b473f40578e726f3e50e219bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:05.714888 4559 raft_consensus.cc:482] T 4d5d4121e8284195b76bfd12f826f942 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:05.715070 4559 raft_consensus.cc:504] T 4d5d4121e8284195b76bfd12f826f942 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:05.715735 4559 leader_election.cc:288] T 4d5d4121e8284195b76bfd12f826f942 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:27:05.716159 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d5d4121e8284195b76bfd12f826f942" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:27:05.716364 31535 raft_consensus.cc:2379] T 4d5d4121e8284195b76bfd12f826f942 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 08d7a0680d244b55b93d17f9ab57b84c in current term 1: Already voted for candidate 9dec645b473f40578e726f3e50e219bf in this term.
I20240502 21:27:05.716588 1964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d5d4121e8284195b76bfd12f826f942" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dec645b473f40578e726f3e50e219bf" is_pre_election: true
I20240502 21:27:05.717101 31730 leader_election.cc:302] T 4d5d4121e8284195b76bfd12f826f942 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c; no voters: 9dec645b473f40578e726f3e50e219bf, a44f41b0b6644fb6aa29476aed8c964f
I20240502 21:27:05.717386 4559 raft_consensus.cc:3037] T 4d5d4121e8284195b76bfd12f826f942 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:05.719910 4559 raft_consensus.cc:2726] T 4d5d4121e8284195b76bfd12f826f942 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:27:05.749091 5810 ts_tablet_manager.cc:902] T 988a0a9425434cd0b724710e9ebf23a6 P 1e04157b98dd4fad98858c290b73c3db: Initiating tablet copy from peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:27:05.756923 31389 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:05.768548 5780 tablet.cc:1722] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:05.769022 5781 tablet.cc:1722] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:05.771476 5782 tablet.cc:1722] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P a44f41b0b6644fb6aa29476aed8c964f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:05.782255 31392 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table_new_name [id=f803fc85ce5940c9b595ddf0fe325413] alter complete (version 3)
I20240502 21:27:05.791407 5810 tablet_copy_client.cc:308] T 988a0a9425434cd0b724710e9ebf23a6 P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.129:41039
I20240502 21:27:05.791954 31556 tablet_copy_service.cc:140] P a44f41b0b6644fb6aa29476aed8c964f: Received BeginTabletCopySession request for tablet 988a0a9425434cd0b724710e9ebf23a6 from peer 1e04157b98dd4fad98858c290b73c3db ({username='slave'} at 127.24.226.133:49939)
I20240502 21:27:05.792079 31556 tablet_copy_service.cc:161] P a44f41b0b6644fb6aa29476aed8c964f: Beginning new tablet copy session on tablet 988a0a9425434cd0b724710e9ebf23a6 from peer 1e04157b98dd4fad98858c290b73c3db at {username='slave'} at 127.24.226.133:49939: session id = 1e04157b98dd4fad98858c290b73c3db-988a0a9425434cd0b724710e9ebf23a6
I20240502 21:27:05.794342 31556 tablet_copy_source_session.cc:205] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:05.795480 5810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 988a0a9425434cd0b724710e9ebf23a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.800678 5810 tablet_copy_client.cc:783] T 988a0a9425434cd0b724710e9ebf23a6 P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:05.800890 5810 tablet_copy_client.cc:647] T 988a0a9425434cd0b724710e9ebf23a6 P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:05.802259 5810 tablet_copy_client.cc:523] T 988a0a9425434cd0b724710e9ebf23a6 P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:05.806356 5810 tablet_bootstrap.cc:492] T 988a0a9425434cd0b724710e9ebf23a6 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:27:05.828157 5522 raft_consensus.cc:482] T 04070a3231524eac83e0c94bf2aee585 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:05.828320 5522 raft_consensus.cc:504] T 04070a3231524eac83e0c94bf2aee585 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:05.828961 5522 leader_election.cc:288] T 04070a3231524eac83e0c94bf2aee585 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:27:05.829682 1834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04070a3231524eac83e0c94bf2aee585" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:27:05.829768 2096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04070a3231524eac83e0c94bf2aee585" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:27:05.829887 1834 raft_consensus.cc:2454] T 04070a3231524eac83e0c94bf2aee585 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:27:05.829985 2096 raft_consensus.cc:2454] T 04070a3231524eac83e0c94bf2aee585 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:27:05.830404 2163 leader_election.cc:302] T 04070a3231524eac83e0c94bf2aee585 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:27:05.830665 5522 raft_consensus.cc:2781] T 04070a3231524eac83e0c94bf2aee585 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:05.830746 5522 raft_consensus.cc:482] T 04070a3231524eac83e0c94bf2aee585 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:05.830830 5522 raft_consensus.cc:3037] T 04070a3231524eac83e0c94bf2aee585 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:05.833259 5522 raft_consensus.cc:504] T 04070a3231524eac83e0c94bf2aee585 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:05.834676 1834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04070a3231524eac83e0c94bf2aee585" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:27:05.834892 1834 raft_consensus.cc:3037] T 04070a3231524eac83e0c94bf2aee585 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:05.835678 5522 leader_election.cc:288] T 04070a3231524eac83e0c94bf2aee585 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Requested vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:27:05.837527 1834 raft_consensus.cc:2454] T 04070a3231524eac83e0c94bf2aee585 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 1.
I20240502 21:27:05.838079 2096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04070a3231524eac83e0c94bf2aee585" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:27:05.838289 2096 raft_consensus.cc:3037] T 04070a3231524eac83e0c94bf2aee585 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:05.840648 2096 raft_consensus.cc:2454] T 04070a3231524eac83e0c94bf2aee585 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 1.
I20240502 21:27:05.841156 2163 leader_election.cc:302] T 04070a3231524eac83e0c94bf2aee585 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:27:05.841451 5661 raft_consensus.cc:2781] T 04070a3231524eac83e0c94bf2aee585 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:05.844842 5661 raft_consensus.cc:686] T 04070a3231524eac83e0c94bf2aee585 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Becoming Leader. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Running, Role: LEADER
I20240502 21:27:05.845139 5661 consensus_queue.cc:237] T 04070a3231524eac83e0c94bf2aee585 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:05.849812 31388 catalog_manager.cc:5484] T 04070a3231524eac83e0c94bf2aee585 P d1406e92aec84ac585304e57bd37a40d reported cstate change: term changed from 0 to 1, leader changed from <none> to d1406e92aec84ac585304e57bd37a40d (127.24.226.136). New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:05.884969 31388 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:05.885177 31388 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:05.888025 31388 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet a6e2e40ab41f4faa80aa4485d4e8e4f6
I20240502 21:27:05.889184 1814 tablet_service.cc:1514] Processing DeleteTablet for tablet a6e2e40ab41f4faa80aa4485d4e8e4f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:05 UTC) from {username='slave'} at 127.0.0.1:47510
I20240502 21:27:05.889245 31775 tablet_service.cc:1514] Processing DeleteTablet for tablet a6e2e40ab41f4faa80aa4485d4e8e4f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:05 UTC) from {username='slave'} at 127.0.0.1:51596
I20240502 21:27:05.889427 5697 tablet_replica.cc:330] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 1e04157b98dd4fad98858c290b73c3db: stopping tablet replica
I20240502 21:27:05.889636 5697 raft_consensus.cc:2229] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:05.889801 5697 raft_consensus.cc:2258] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:05.890635 5697 ts_tablet_manager.cc:1875] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 1e04157b98dd4fad98858c290b73c3db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:05.892872 31516 tablet_service.cc:1514] Processing DeleteTablet for tablet a6e2e40ab41f4faa80aa4485d4e8e4f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:05 UTC) from {username='slave'} at 127.0.0.1:45194
I20240502 21:27:05.899202 5697 ts_tablet_manager.cc:1888] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 1e04157b98dd4fad98858c290b73c3db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:05.899370 5697 log.cc:1196] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 1e04157b98dd4fad98858c290b73c3db: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal/wals/a6e2e40ab41f4faa80aa4485d4e8e4f6
I20240502 21:27:05.899919 5697 ts_tablet_manager.cc:1909] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 1e04157b98dd4fad98858c290b73c3db: Deleting consensus metadata
I20240502 21:27:05.901923 31381 catalog_manager.cc:4830] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): tablet a6e2e40ab41f4faa80aa4485d4e8e4f6 (table rebalancer_dynamic_table_new_name [id=f803fc85ce5940c9b595ddf0fe325413]) successfully deleted
I20240502 21:27:05.902477 5830 tablet_replica.cc:330] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:27:05.902789 5810 tablet_bootstrap.cc:492] T 988a0a9425434cd0b724710e9ebf23a6 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:05.902762 5830 raft_consensus.cc:2229] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:05.903070 5830 raft_consensus.cc:2258] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:05.903164 5810 tablet_bootstrap.cc:492] T 988a0a9425434cd0b724710e9ebf23a6 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap complete.
I20240502 21:27:05.903396 5810 ts_tablet_manager.cc:1367] T 988a0a9425434cd0b724710e9ebf23a6 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.097s user 0.048s sys 0.005s
I20240502 21:27:05.904078 5810 raft_consensus.cc:348] T 988a0a9425434cd0b724710e9ebf23a6 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } }
I20240502 21:27:05.904325 5810 raft_consensus.cc:729] T 988a0a9425434cd0b724710e9ebf23a6 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: LEARNER
I20240502 21:27:05.904260 5830 ts_tablet_manager.cc:1875] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:05.904542 5810 consensus_queue.cc:260] T 988a0a9425434cd0b724710e9ebf23a6 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } }
I20240502 21:27:05.905169 5810 ts_tablet_manager.cc:1398] T 988a0a9425434cd0b724710e9ebf23a6 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:05.906082 31556 tablet_copy_service.cc:342] P a44f41b0b6644fb6aa29476aed8c964f: Request end of tablet copy session 1e04157b98dd4fad98858c290b73c3db-988a0a9425434cd0b724710e9ebf23a6 received from {username='slave'} at 127.24.226.133:49939
I20240502 21:27:05.906188 31556 tablet_copy_service.cc:433] P a44f41b0b6644fb6aa29476aed8c964f: ending tablet copy session 1e04157b98dd4fad98858c290b73c3db-988a0a9425434cd0b724710e9ebf23a6 on tablet 988a0a9425434cd0b724710e9ebf23a6 with peer 1e04157b98dd4fad98858c290b73c3db
I20240502 21:27:05.914961 5831 tablet_replica.cc:330] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c: stopping tablet replica
I20240502 21:27:05.915360 5831 raft_consensus.cc:2229] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:05.915831 5831 raft_consensus.cc:2258] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:05.917402 5831 ts_tablet_manager.cc:1875] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:05.918025 5830 ts_tablet_manager.cc:1888] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:05.918186 5830 log.cc:1196] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/a6e2e40ab41f4faa80aa4485d4e8e4f6
I20240502 21:27:05.918530 5830 ts_tablet_manager.cc:1909] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P a44f41b0b6644fb6aa29476aed8c964f: Deleting consensus metadata
I20240502 21:27:05.919297 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet a6e2e40ab41f4faa80aa4485d4e8e4f6 (table rebalancer_dynamic_table_new_name [id=f803fc85ce5940c9b595ddf0fe325413]) successfully deleted
I20240502 21:27:05.922516 5831 ts_tablet_manager.cc:1888] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:05.922684 5831 log.cc:1196] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/wals/a6e2e40ab41f4faa80aa4485d4e8e4f6
I20240502 21:27:05.923084 5831 ts_tablet_manager.cc:1909] T a6e2e40ab41f4faa80aa4485d4e8e4f6 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting consensus metadata
I20240502 21:27:05.924317 31381 catalog_manager.cc:4830] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): tablet a6e2e40ab41f4faa80aa4485d4e8e4f6 (table rebalancer_dynamic_table_new_name [id=f803fc85ce5940c9b595ddf0fe325413]) successfully deleted
I20240502 21:27:05.991050 31389 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:05.999724 1814 tablet_service.cc:1467] Processing CreateTablet for tablet 5040da1f38a24860bfd89637cdeab4e7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ca5e0c1059364d20a3e8e174f4ed6fa3]), partition=
I20240502 21:27:06.000108 1814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5040da1f38a24860bfd89637cdeab4e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:06.001302 1945 tablet_service.cc:1467] Processing CreateTablet for tablet 5040da1f38a24860bfd89637cdeab4e7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ca5e0c1059364d20a3e8e174f4ed6fa3]), partition=
I20240502 21:27:06.001638 1945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5040da1f38a24860bfd89637cdeab4e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:06.001444 1834 consensus_queue.cc:237] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } }
I20240502 21:27:06.007458 31906 tablet_service.cc:1467] Processing CreateTablet for tablet 5040da1f38a24860bfd89637cdeab4e7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ca5e0c1059364d20a3e8e174f4ed6fa3]), partition=
I20240502 21:27:06.007833 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5040da1f38a24860bfd89637cdeab4e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:06.017000 5470 tablet_bootstrap.cc:492] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:27:06.018396 5470 tablet_bootstrap.cc:654] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:06.021142 2096 raft_consensus.cc:1261] T 04c07a003bfe472e9142df6082759ae1 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:06.022321 1964 raft_consensus.cc:1261] T 04c07a003bfe472e9142df6082759ae1 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:06.022496 5874 tablet_bootstrap.cc:492] T 5040da1f38a24860bfd89637cdeab4e7 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:27:06.023073 5854 consensus_queue.cc:1035] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:06.023618 5854 consensus_queue.cc:1035] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:27:06.027565 1770 consensus_peers.cc:485] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db -> Peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): Couldn't send request to peer 05f4fac41c4f4959b9b316cb112d0ada. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04c07a003bfe472e9142df6082759ae1. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:06.027909 5537 consensus_queue.cc:1035] T 78f06bc68b954dc4b55b3c8c3c612c2f P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:06.031024 5874 tablet_bootstrap.cc:654] T 5040da1f38a24860bfd89637cdeab4e7 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:06.043056 5470 tablet_bootstrap.cc:492] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:27:06.043318 5470 ts_tablet_manager.cc:1367] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.008s
I20240502 21:27:06.043941 5874 tablet_bootstrap.cc:492] T 5040da1f38a24860bfd89637cdeab4e7 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:27:06.044132 5470 raft_consensus.cc:348] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:06.044348 5470 raft_consensus.cc:374] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:06.044416 5470 raft_consensus.cc:729] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:27:06.044692 5470 consensus_queue.cc:260] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:06.044831 5874 ts_tablet_manager.cc:1367] T 5040da1f38a24860bfd89637cdeab4e7 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.000s
I20240502 21:27:06.045291 5470 ts_tablet_manager.cc:1398] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:06.046243 5874 raft_consensus.cc:348] T 5040da1f38a24860bfd89637cdeab4e7 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:06.047835 5874 raft_consensus.cc:374] T 5040da1f38a24860bfd89637cdeab4e7 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:06.047940 5874 raft_consensus.cc:729] T 5040da1f38a24860bfd89637cdeab4e7 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:27:06.048194 5874 consensus_queue.cc:260] T 5040da1f38a24860bfd89637cdeab4e7 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:06.051286 2096 raft_consensus.cc:2932] T 04c07a003bfe472e9142df6082759ae1 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } } }
I20240502 21:27:06.051376 5854 raft_consensus.cc:2932] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } } }
I20240502 21:27:06.057844 5874 ts_tablet_manager.cc:1398] T 5040da1f38a24860bfd89637cdeab4e7 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240502 21:27:06.058475 31392 catalog_manager.cc:5484] T 04c07a003bfe472e9142df6082759ae1 P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } } }
I20240502 21:27:06.062167 5882 ts_tablet_manager.cc:902] T 043f33cb58084c3ebbce4893d1b63fbc P 9dec645b473f40578e726f3e50e219bf: Initiating tablet copy from peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
F20240502 21:27:06.062530 5882 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-043f33cb58084c3ebbce4893d1b63fbc.queue_time_us, tablet-download-pool-043f33cb58084c3ebbce4893d1b63fbc.queue_time_us) (tablet-download-pool-0b8d119ee8474a40ac08df61df5b8423.queue_time_us, tablet-download-pool-0b8d119ee8474a40ac08df61df5b8423.queue_time_us) (tablet-download-pool-0b8d119ee8474a40ac08df61df5b8423.run_cpu_time_us, tablet-download-pool-0b8d119ee8474a40ac08df61df5b8423.run_cpu_time_us) (tablet-download-pool-0b8d119ee8474a40ac08df61df5b8423.run_wall_time_us, tablet-download-pool-0b8d119ee8474a40ac08df61df5b8423.run_wall_time_us) (tablet-download-pool-182b3f15c06c4c07ade0a8854d3ecafd.queue_time_us, tablet-download-pool-182b3f15c06c4c07ade0a8854d3ecafd.queue_time_us) (tablet-download-pool-182b3f15c06c4c07ade0a8854d3ecafd.run_cpu_time_us, tablet-download-pool-182b3f15c06c4c07ade0a8854d3ecafd.run_cpu_time_us) (tablet-download-pool-182b3f15c06c4c07ade0a8854d3ecafd.run_wall_time_us, tablet-download-pool-182b3f15c06c4c07ade0a8854d3ecafd.run_wall_time_us) (tablet-download-pool-3e6657723fe84e749dea768de44638e7.queue_time_us, tablet-download-pool-3e6657723fe84e749dea768de44638e7.queue_time_us) (tablet-download-pool-3e6657723fe84e749dea768de44638e7.run_cpu_time_us, tablet-download-pool-3e6657723fe84e749dea768de44638e7.run_cpu_time_us) (tablet-download-pool-3e6657723fe84e749dea768de44638e7.run_wall_time_us, tablet-download-pool-3e6657723fe84e749dea768de44638e7.run_wall_time_us) (tablet-download-pool-55fc18e4cc98446ca50c1dd2b6a9a89d.queue_time_us, tablet-download-pool-55fc18e4cc98446ca50c1dd2b6a9a89d.queue_time_us) (tablet-download-pool-55fc18e4cc98446ca50c1dd2b6a9a89d.run_cpu_time_us, tablet-download-pool-55fc18e4cc98446ca50c1dd2b6a9a89d.run_cpu_time_us) (tablet-download-pool-55fc18e4cc98446ca50c1dd2b6a9a89d.run_wall_time_us, tablet-download-pool-55fc18e4cc98446ca50c1dd2b6a9a89d.run_wall_time_us) (tablet-download-pool-59f36d947f4341f39b6969cc1e065b6c.queue_time_us, tablet-download-pool-59f36d947f4341f39b6969cc1e065b6c.queue_time_us) (tablet-download-pool-59f36d947f4341f39b6969cc1e065b6c.run_cpu_time_us, tablet-download-pool-59f36d947f4341f39b6969cc1e065b6c.run_cpu_time_us) (tablet-download-pool-59f36d947f4341f39b6969cc1e065b6c.run_wall_time_us, tablet-download-pool-59f36d947f4341f39b6969cc1e065b6c.run_wall_time_us) (tablet-download-pool-5bc6c7211d5f4f73b2a050b9ec78c6b1.queue_time_us, tablet-download-pool-5bc6c7211d5f4f73b2a050b9ec78c6b1.queue_time_us) (tablet-download-pool-5bc6c7211d5f4f73b2a050b9ec78c6b1.run_cpu_time_us, tablet-download-pool-5bc6c7211d5f4f73b2a050b9ec78c6b1.run_cpu_time_us) (tablet-download-pool-5bc6c7211d5f4f73b2a050b9ec78c6b1.run_wall_time_us, tablet-download-pool-5bc6c7211d5f4f73b2a050b9ec78c6b1.run_wall_time_us) (tablet-download-pool-62f083e9641042c990efcfb1c4104de5.queue_time_us, tablet-download-pool-62f083e9641042c990efcfb1c4104de5.queue_time_us) (tablet-download-pool-62f083e9641042c990efcfb1c4104de5.run_cpu_time_us, tablet-download-pool-62f083e9641042c990efcfb1c4104de5.run_cpu_time_us) (tablet-download-pool-62f083e9641042c990efcfb1c4104de5.run_wall_time_us, tablet-download-pool-62f083e9641042c990efcfb1c4104de5.run_wall_time_us) (tablet-download-pool-759b7529d7d84baab1a4622d34f803bf.queue_time_us, tablet-download-pool-759b7529d7d84baab1a4622d34f803bf.queue_time_us) (tablet-download-pool-759b7529d7d84baab1a4622d34f803bf.run_cpu_time_us, tablet-download-pool-759b7529d7d84baab1a4622d34f803bf.run_cpu_time_us) (tablet-download-pool-759b7529d7d84baab1a4622d34f803bf.run_wall_time_us, tablet-download-pool-759b7529d7d84baab1a4622d34f803bf.run_wall_time_us) (tablet-download-pool-ae56d18c629a4e5eb50911c89de4227c.queue_time_us, tablet-download-pool-ae56d18c629a4e5eb50911c89de4227c.queue_time_us) (tablet-download-pool-ae56d18c629a4e5eb50911c89de4227c.run_cpu_time_us, tablet-download-pool-ae56d18c629a4e5eb50911c89de4227c.run_cpu_time_us) (tablet-download-pool-ae56d18c629a4e5eb50911c89de4227c.run_wall_time_us, tablet-download-pool-ae56d18c629a4e5eb50911c89de4227c.run_wall_time_us) (tablet-download-pool-b407c2c9b3a14ce3ada52b7856c6ecee.queue_time_us, tablet-download-pool-b407c2c9b3a14ce3ada52b7856c6ecee.queue_time_us) (tablet-download-pool-b407c2c9b3a14ce3ada52b7856c6ecee.run_cpu_time_us, tablet-download-pool-b407c2c9b3a14ce3ada52b7856c6ecee.run_cpu_time_us) (tablet-download-pool-b407c2c9b3a14ce3ada52b7856c6ecee.run_wall_time_us, tablet-download-pool-b407c2c9b3a14ce3ada52b7856c6ecee.run_wall_time_us) (tablet-download-pool-b9342dbe0497499ba4c238f2ed033293.queue_time_us, tablet-download-pool-b9342dbe0497499ba4c238f2ed033293.queue_time_us) (tablet-download-pool-b9342dbe0497499ba4c238f2ed033293.run_cpu_time_us, tablet-download-pool-b9342dbe0497499ba4c238f2ed033293.run_cpu_time_us) (tablet-download-pool-b9342dbe0497499ba4c238f2ed033293.run_wall_time_us, tablet-download-pool-b9342dbe0497499ba4c238f2ed033293.run_wall_time_us) (tablet-download-pool-bb64e0d78397429d959535f570658594.queue_time_us, tablet-download-pool-bb64e0d78397429d959535f570658594.queue_time_us) (tablet-download-pool-bb64e0d78397429d959535f570658594.run_cpu_time_us, tablet-download-pool-bb64e0d78397429d959535f570658594.run_cpu_time_us) (tablet-download-pool-bb64e0d78397429d959535f570658594.run_wall_time_us, tablet-download-pool-bb64e0d78397429d959535f570658594.run_wall_time_us) (tablet-download-pool-f9bbc56ba2f841b9bb36a864e1a1f808.queue_time_us, tablet-download-pool-f9bbc56ba2f841b9bb36a864e1a1f808.queue_time_us) (tablet-download-pool-f9bbc56ba2f841b9bb36a864e1a1f808.run_cpu_time_us, tablet-download-pool-f9bbc56ba2f841b9bb36a864e1a1f808.run_cpu_time_us) (tablet-download-pool-f9bbc56ba2f841b9bb36a864e1a1f808.run_wall_time_us, tablet-download-pool-f9bbc56ba2f841b9bb36a864e1a1f808.run_wall_time_us) (tablet-download-pool-fc93b5adc1d94e809e97e56b56822611.queue_time_us, tablet-download-pool-fc93b5adc1d94e809e97e56b56822611.queue_time_us) (tablet-download-pool-fc93b5adc1d94e809e97e56b56822611.run_cpu_time_us, tablet-download-pool-fc93b5adc1d94e809e97e56b56822611.run_cpu_time_us) (tablet-download-pool-fc93b5adc1d94e809e97e56b56822611.run_wall_time_us, tablet-download-pool-fc93b5adc1d94e809e97e56b56822611.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f81e147fdcd google::LogMessage::Fail() at ??:0
@ 0x7f81e1483b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f81e147f7cc google::LogMessage::Flush() at ??:0
I20240502 21:27:06.070066 5787 consensus_queue.cc:1035] T 4d5d4121e8284195b76bfd12f826f942 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
@ 0x7f81e1480f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f81e41c0742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f81e418cf8b kudu::ThreadPool::ThreadPool() at ??:0
I20240502 21:27:06.080992 5873 tablet_bootstrap.cc:492] T 5040da1f38a24860bfd89637cdeab4e7 P 9dec645b473f40578e726f3e50e219bf: Bootstrap starting.
@ 0x7f81e4186cd6 kudu::ThreadPoolBuilder::Build() at ??:0
I20240502 21:27:06.085184 5873 tablet_bootstrap.cc:654] T 5040da1f38a24860bfd89637cdeab4e7 P 9dec645b473f40578e726f3e50e219bf: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:06.086231 5345 consensus_queue.cc:1035] T 78f06bc68b954dc4b55b3c8c3c612c2f P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
@ 0x7f81f35d98de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f81f35f1daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240502 21:27:06.099938 1965 raft_consensus.cc:2932] T 04c07a003bfe472e9142df6082759ae1 P 9dec645b473f40578e726f3e50e219bf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } } }
@ 0x7f81f371c480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240502 21:27:06.115300 2096 raft_consensus.cc:1203] T 62e70b8e69c94d76b23c655a3f5de5bd P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
@ 0x7f81f3754a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f81f3737981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
I20240502 21:27:06.134953 5345 consensus_queue.cc:1035] T 4d5d4121e8284195b76bfd12f826f942 P 9dec645b473f40578e726f3e50e219bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
@ 0x7f81f373770e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f81e4190b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f81e419419c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f81e4194019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f81e416f01e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f81e74f46db start_thread at ??:0
@ 0x7f81e033271f clone at ??:0
I20240502 21:27:06.193827 31924 consensus_queue.cc:237] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } }
I20240502 21:27:06.204810 31797 raft_consensus.cc:1261] T 88189733d9a14560971dbeb71543073f P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:06.205286 2096 raft_consensus.cc:1261] T 88189733d9a14560971dbeb71543073f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:06.205562 5896 consensus_queue.cc:1035] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:06.205871 5512 consensus_queue.cc:1035] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:06.217974 5643 raft_consensus.cc:482] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:06.218186 5643 raft_consensus.cc:504] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:06.219290 5643 leader_election.cc:288] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:27:06.223848 31924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5040da1f38a24860bfd89637cdeab4e7" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:27:06.224032 31924 raft_consensus.cc:2454] T 5040da1f38a24860bfd89637cdeab4e7 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 0.
I20240502 21:27:06.225026 5512 raft_consensus.cc:2932] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:27:06.226020 5661 consensus_queue.cc:1035] T 04070a3231524eac83e0c94bf2aee585 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:06.227413 1770 leader_election.cc:302] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:27:06.229261 2093 raft_consensus.cc:2932] T 88189733d9a14560971dbeb71543073f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:27:06.230201 5643 raft_consensus.cc:2781] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:06.230314 5643 raft_consensus.cc:482] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:06.230485 5643 raft_consensus.cc:3037] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:06.234547 5643 raft_consensus.cc:504] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:06.237015 31924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5040da1f38a24860bfd89637cdeab4e7" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:27:06.237221 31924 raft_consensus.cc:3037] T 5040da1f38a24860bfd89637cdeab4e7 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:06.240097 31924 raft_consensus.cc:2454] T 5040da1f38a24860bfd89637cdeab4e7 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:27:06.240828 1770 leader_election.cc:302] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:27:06.241084 5215 raft_consensus.cc:2781] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:06.241257 5215 raft_consensus.cc:686] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Becoming Leader. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Running, Role: LEADER
I20240502 21:27:06.241510 5215 consensus_queue.cc:237] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:06.241878 5643 leader_election.cc:288] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Requested vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:27:06.242311 31797 raft_consensus.cc:2932] T 88189733d9a14560971dbeb71543073f P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } } }
I20240502 21:27:06.245616 31392 catalog_manager.cc:5484] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: config changed from index 6 to 7, NON_VOTER 9dec645b473f40578e726f3e50e219bf (127.24.226.134) added. New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: NON_VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:06.251209 5661 consensus_queue.cc:1035] T 04070a3231524eac83e0c94bf2aee585 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:06.267062 31392 catalog_manager.cc:5484] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e04157b98dd4fad98858c290b73c3db (127.24.226.133). New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:06.308387 31797 consensus_queue.cc:237] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } }
I20240502 21:27:06.311043 2225 raft_consensus.cc:1261] T 5a54d2aad63c4f3182992f00b8564acf P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:06.311869 5417 consensus_queue.cc:1035] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 21:27:06.312634 31732 consensus_peers.cc:485] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c -> Peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): Couldn't send request to peer 1e04157b98dd4fad98858c290b73c3db. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a54d2aad63c4f3182992f00b8564acf. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:06.313673 31924 raft_consensus.cc:1261] T 5a54d2aad63c4f3182992f00b8564acf P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:06.314680 5417 consensus_queue.cc:1035] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:06.328677 5417 raft_consensus.cc:2932] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } } }
I20240502 21:27:06.332233 31925 raft_consensus.cc:2932] T 5a54d2aad63c4f3182992f00b8564acf P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } } }
I20240502 21:27:06.343027 2225 raft_consensus.cc:2932] T 5a54d2aad63c4f3182992f00b8564acf P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } } }
I20240502 21:27:06.343241 31389 catalog_manager.cc:5484] T 5a54d2aad63c4f3182992f00b8564acf P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: config changed from index 6 to 7, NON_VOTER 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) added. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } } }
I20240502 21:27:06.356914 1834 raft_consensus.cc:1203] T 988a0a9425434cd0b724710e9ebf23a6 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:06.417912 31389 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:06.429697 31924 raft_consensus.cc:1261] T 5040da1f38a24860bfd89637cdeab4e7 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:06.430280 5215 consensus_queue.cc:1035] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:06.430595 5929 ts_tablet_manager.cc:902] T 04c07a003bfe472e9142df6082759ae1 P 05f4fac41c4f4959b9b316cb112d0ada: Initiating tablet copy from peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:27:06.446128 31389 catalog_manager.cc:5576] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933) does not have the latest schema for tablet 5040da1f38a24860bfd89637cdeab4e7 (table rebalancer_dynamic_table [id=ca5e0c1059364d20a3e8e174f4ed6fa3]). Expected version 1 got 0
I20240502 21:27:06.459889 5934 tablet.cc:1722] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:06.460351 5934 tablet.cc:1620] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db: MemRowSet was empty: no flush needed.
I20240502 21:27:06.460955 5935 tablet.cc:1722] T 5040da1f38a24860bfd89637cdeab4e7 P 2d50f33557944bb480ea32199d9ee4f1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:06.461375 5935 tablet.cc:1620] T 5040da1f38a24860bfd89637cdeab4e7 P 2d50f33557944bb480ea32199d9ee4f1: MemRowSet was empty: no flush needed.
I20240502 21:27:06.466214 31392 catalog_manager.cc:5576] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933) does not have the latest schema for tablet 5040da1f38a24860bfd89637cdeab4e7 (table rebalancer_dynamic_table [id=ca5e0c1059364d20a3e8e174f4ed6fa3]). Expected version 1 got 0
I20240502 21:27:06.466256 31388 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_extra_table_46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:06.477829 31389 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=ca5e0c1059364d20a3e8e174f4ed6fa3] alter complete (version 1)
I20240502 21:27:06.482764 5929 tablet_copy_client.cc:308] T 04c07a003bfe472e9142df6082759ae1 P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.133:39863
I20240502 21:27:06.487499 1854 tablet_copy_service.cc:140] P 1e04157b98dd4fad98858c290b73c3db: Received BeginTabletCopySession request for tablet 04c07a003bfe472e9142df6082759ae1 from peer 05f4fac41c4f4959b9b316cb112d0ada ({username='slave'} at 127.24.226.130:43577)
I20240502 21:27:06.487646 1854 tablet_copy_service.cc:161] P 1e04157b98dd4fad98858c290b73c3db: Beginning new tablet copy session on tablet 04c07a003bfe472e9142df6082759ae1 from peer 05f4fac41c4f4959b9b316cb112d0ada at {username='slave'} at 127.24.226.130:43577: session id = 05f4fac41c4f4959b9b316cb112d0ada-04c07a003bfe472e9142df6082759ae1
I20240502 21:27:06.490782 1854 tablet_copy_source_session.cc:205] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:06.491798 31645 tablet_service.cc:1467] Processing CreateTablet for tablet eed5a5c41d9a46ce8a712b5b399e0472 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=f8144a30dfd04c19a80430f596b4a89d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:06.492744 31645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eed5a5c41d9a46ce8a712b5b399e0472. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:06.493788 2077 tablet_service.cc:1467] Processing CreateTablet for tablet 6c84ffd75b4048d886210d6ecbe4d717 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=f8144a30dfd04c19a80430f596b4a89d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:06.494136 2076 tablet_service.cc:1467] Processing CreateTablet for tablet df919f52065f444fbc2e9fb59a0494a5 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=f8144a30dfd04c19a80430f596b4a89d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:06.494266 2077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c84ffd75b4048d886210d6ecbe4d717. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:06.494542 2076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df919f52065f444fbc2e9fb59a0494a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:06.497061 31905 tablet_service.cc:1467] Processing CreateTablet for tablet df919f52065f444fbc2e9fb59a0494a5 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=f8144a30dfd04c19a80430f596b4a89d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:06.497450 31905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df919f52065f444fbc2e9fb59a0494a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:06.497769 31516 tablet_service.cc:1467] Processing CreateTablet for tablet eed5a5c41d9a46ce8a712b5b399e0472 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=f8144a30dfd04c19a80430f596b4a89d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:06.497879 31514 tablet_service.cc:1467] Processing CreateTablet for tablet df919f52065f444fbc2e9fb59a0494a5 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=f8144a30dfd04c19a80430f596b4a89d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:06.498132 31516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eed5a5c41d9a46ce8a712b5b399e0472. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:06.498360 31514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df919f52065f444fbc2e9fb59a0494a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:06.501260 2208 tablet_service.cc:1467] Processing CreateTablet for tablet 6c84ffd75b4048d886210d6ecbe4d717 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=f8144a30dfd04c19a80430f596b4a89d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:06.502933 5874 tablet_bootstrap.cc:492] T df919f52065f444fbc2e9fb59a0494a5 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:27:06.497061 31906 tablet_service.cc:1467] Processing CreateTablet for tablet 6c84ffd75b4048d886210d6ecbe4d717 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=f8144a30dfd04c19a80430f596b4a89d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:06.503278 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c84ffd75b4048d886210d6ecbe4d717. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:06.504365 5874 tablet_bootstrap.cc:654] T df919f52065f444fbc2e9fb59a0494a5 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:06.511098 5929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04c07a003bfe472e9142df6082759ae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:06.512662 5874 tablet_bootstrap.cc:492] T df919f52065f444fbc2e9fb59a0494a5 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:27:06.512807 5874 ts_tablet_manager.cc:1367] T df919f52065f444fbc2e9fb59a0494a5 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240502 21:27:06.513896 5874 raft_consensus.cc:348] T df919f52065f444fbc2e9fb59a0494a5 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:06.514118 5874 raft_consensus.cc:374] T df919f52065f444fbc2e9fb59a0494a5 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:06.514175 5874 raft_consensus.cc:729] T df919f52065f444fbc2e9fb59a0494a5 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:27:06.501674 2208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c84ffd75b4048d886210d6ecbe4d717. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:06.514446 5874 consensus_queue.cc:260] T df919f52065f444fbc2e9fb59a0494a5 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:06.515100 5874 ts_tablet_manager.cc:1398] T df919f52065f444fbc2e9fb59a0494a5 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:06.517498 5944 tablet_bootstrap.cc:492] T eed5a5c41d9a46ce8a712b5b399e0472 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:27:06.518536 5874 tablet_bootstrap.cc:492] T 6c84ffd75b4048d886210d6ecbe4d717 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:27:06.519383 5944 tablet_bootstrap.cc:654] T eed5a5c41d9a46ce8a712b5b399e0472 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:06.519966 5874 tablet_bootstrap.cc:654] T 6c84ffd75b4048d886210d6ecbe4d717 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:06.529439 5874 tablet_bootstrap.cc:492] T 6c84ffd75b4048d886210d6ecbe4d717 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:27:06.529574 5874 ts_tablet_manager.cc:1367] T 6c84ffd75b4048d886210d6ecbe4d717 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240502 21:27:06.530391 5874 raft_consensus.cc:348] T 6c84ffd75b4048d886210d6ecbe4d717 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:06.530697 5874 raft_consensus.cc:374] T 6c84ffd75b4048d886210d6ecbe4d717 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:06.530761 5874 raft_consensus.cc:729] T 6c84ffd75b4048d886210d6ecbe4d717 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:27:06.531002 5874 consensus_queue.cc:260] T 6c84ffd75b4048d886210d6ecbe4d717 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:06.531895 5944 tablet_bootstrap.cc:492] T eed5a5c41d9a46ce8a712b5b399e0472 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:27:06.532003 5944 ts_tablet_manager.cc:1367] T eed5a5c41d9a46ce8a712b5b399e0472 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.004s
I20240502 21:27:06.532140 5929 tablet_copy_client.cc:783] T 04c07a003bfe472e9142df6082759ae1 P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:06.532351 5929 tablet_copy_client.cc:647] T 04c07a003bfe472e9142df6082759ae1 P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:06.535109 5874 ts_tablet_manager.cc:1398] T 6c84ffd75b4048d886210d6ecbe4d717 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:27:06.534935 5944 raft_consensus.cc:348] T eed5a5c41d9a46ce8a712b5b399e0472 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:06.535252 5944 raft_consensus.cc:374] T eed5a5c41d9a46ce8a712b5b399e0472 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:06.535305 5944 raft_consensus.cc:729] T eed5a5c41d9a46ce8a712b5b399e0472 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:27:06.535491 5944 consensus_queue.cc:260] T eed5a5c41d9a46ce8a712b5b399e0472 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:06.536012 5944 ts_tablet_manager.cc:1398] T eed5a5c41d9a46ce8a712b5b399e0472 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:06.536233 5929 tablet_copy_client.cc:523] T 04c07a003bfe472e9142df6082759ae1 P 05f4fac41c4f4959b9b316cb112d0ada: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:06.539113 5941 tablet_bootstrap.cc:492] T 6c84ffd75b4048d886210d6ecbe4d717 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:27:06.539614 5929 tablet_bootstrap.cc:492] T 04c07a003bfe472e9142df6082759ae1 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:27:06.542433 5941 tablet_bootstrap.cc:654] T 6c84ffd75b4048d886210d6ecbe4d717 P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:06.552372 5944 tablet_bootstrap.cc:492] T df919f52065f444fbc2e9fb59a0494a5 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:27:06.555178 5948 tablet_bootstrap.cc:492] T eed5a5c41d9a46ce8a712b5b399e0472 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:27:06.557534 5944 tablet_bootstrap.cc:654] T df919f52065f444fbc2e9fb59a0494a5 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:06.559571 5949 tablet_bootstrap.cc:492] T 6c84ffd75b4048d886210d6ecbe4d717 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:27:06.560863 5941 tablet_bootstrap.cc:492] T 6c84ffd75b4048d886210d6ecbe4d717 P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:27:06.561031 5941 ts_tablet_manager.cc:1367] T 6c84ffd75b4048d886210d6ecbe4d717 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.000s
I20240502 21:27:06.562351 5941 raft_consensus.cc:348] T 6c84ffd75b4048d886210d6ecbe4d717 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:06.563074 5941 raft_consensus.cc:374] T 6c84ffd75b4048d886210d6ecbe4d717 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:06.563191 5941 raft_consensus.cc:729] T 6c84ffd75b4048d886210d6ecbe4d717 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:27:06.563465 5941 consensus_queue.cc:260] T 6c84ffd75b4048d886210d6ecbe4d717 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:06.567039 5948 tablet_bootstrap.cc:654] T eed5a5c41d9a46ce8a712b5b399e0472 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:06.568019 5949 tablet_bootstrap.cc:654] T 6c84ffd75b4048d886210d6ecbe4d717 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:06.569237 5929 tablet_bootstrap.cc:492] T 04c07a003bfe472e9142df6082759ae1 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:06.569604 5929 tablet_bootstrap.cc:492] T 04c07a003bfe472e9142df6082759ae1 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap complete.
I20240502 21:27:06.569845 5929 ts_tablet_manager.cc:1367] T 04c07a003bfe472e9142df6082759ae1 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.030s user 0.012s sys 0.015s
I20240502 21:27:06.573545 5929 raft_consensus.cc:348] T 04c07a003bfe472e9142df6082759ae1 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } }
I20240502 21:27:06.573835 5929 raft_consensus.cc:729] T 04c07a003bfe472e9142df6082759ae1 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: LEARNER
I20240502 21:27:06.574038 5929 consensus_queue.cc:260] T 04c07a003bfe472e9142df6082759ae1 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: NON_VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: true } }
I20240502 21:27:06.574869 5929 ts_tablet_manager.cc:1398] T 04c07a003bfe472e9142df6082759ae1 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:27:06.575656 5948 tablet_bootstrap.cc:492] T eed5a5c41d9a46ce8a712b5b399e0472 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:27:06.575795 5948 ts_tablet_manager.cc:1367] T eed5a5c41d9a46ce8a712b5b399e0472 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.003s
I20240502 21:27:06.576665 5948 raft_consensus.cc:348] T eed5a5c41d9a46ce8a712b5b399e0472 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:06.576848 5948 raft_consensus.cc:374] T eed5a5c41d9a46ce8a712b5b399e0472 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:06.576917 5948 raft_consensus.cc:729] T eed5a5c41d9a46ce8a712b5b399e0472 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:27:06.577167 5948 consensus_queue.cc:260] T eed5a5c41d9a46ce8a712b5b399e0472 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:06.577844 5948 ts_tablet_manager.cc:1398] T eed5a5c41d9a46ce8a712b5b399e0472 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:06.578105 1854 tablet_copy_service.cc:342] P 1e04157b98dd4fad98858c290b73c3db: Request end of tablet copy session 05f4fac41c4f4959b9b316cb112d0ada-04c07a003bfe472e9142df6082759ae1 received from {username='slave'} at 127.24.226.130:43577
I20240502 21:27:06.578254 1854 tablet_copy_service.cc:433] P 1e04157b98dd4fad98858c290b73c3db: ending tablet copy session 05f4fac41c4f4959b9b316cb112d0ada-04c07a003bfe472e9142df6082759ae1 on tablet 04c07a003bfe472e9142df6082759ae1 with peer 05f4fac41c4f4959b9b316cb112d0ada
I20240502 21:27:06.580327 5944 tablet_bootstrap.cc:492] T df919f52065f444fbc2e9fb59a0494a5 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:27:06.580461 5944 ts_tablet_manager.cc:1367] T df919f52065f444fbc2e9fb59a0494a5 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20240502 21:27:06.581041 5944 raft_consensus.cc:348] T df919f52065f444fbc2e9fb59a0494a5 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:06.581218 5944 raft_consensus.cc:374] T df919f52065f444fbc2e9fb59a0494a5 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:06.581261 5944 raft_consensus.cc:729] T df919f52065f444fbc2e9fb59a0494a5 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:27:06.581421 5944 consensus_queue.cc:260] T df919f52065f444fbc2e9fb59a0494a5 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:06.583159 5944 ts_tablet_manager.cc:1398] T df919f52065f444fbc2e9fb59a0494a5 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:27:06.590469 5949 tablet_bootstrap.cc:492] T 6c84ffd75b4048d886210d6ecbe4d717 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:27:06.590646 5949 ts_tablet_manager.cc:1367] T 6c84ffd75b4048d886210d6ecbe4d717 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.000s
I20240502 21:27:06.591398 5949 raft_consensus.cc:348] T 6c84ffd75b4048d886210d6ecbe4d717 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:06.591619 5949 raft_consensus.cc:374] T 6c84ffd75b4048d886210d6ecbe4d717 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:06.591714 5949 raft_consensus.cc:729] T 6c84ffd75b4048d886210d6ecbe4d717 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:27:06.591931 5949 consensus_queue.cc:260] T 6c84ffd75b4048d886210d6ecbe4d717 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:06.592465 5949 ts_tablet_manager.cc:1398] T 6c84ffd75b4048d886210d6ecbe4d717 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:06.592670 5941 ts_tablet_manager.cc:1398] T 6c84ffd75b4048d886210d6ecbe4d717 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.029s user 0.004s sys 0.000s
I20240502 21:27:06.596876 31392 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:06.599408 5941 tablet_bootstrap.cc:492] T df919f52065f444fbc2e9fb59a0494a5 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:27:06.606822 5941 tablet_bootstrap.cc:654] T df919f52065f444fbc2e9fb59a0494a5 P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:06.612881 5934 tablet.cc:1722] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:06.613459 5934 tablet.cc:1620] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db: MemRowSet was empty: no flush needed.
I20240502 21:27:06.614867 5935 tablet.cc:1722] T 5040da1f38a24860bfd89637cdeab4e7 P 2d50f33557944bb480ea32199d9ee4f1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:06.615496 5935 tablet.cc:1620] T 5040da1f38a24860bfd89637cdeab4e7 P 2d50f33557944bb480ea32199d9ee4f1: MemRowSet was empty: no flush needed.
I20240502 21:27:06.621773 5941 tablet_bootstrap.cc:492] T df919f52065f444fbc2e9fb59a0494a5 P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:27:06.622033 5941 ts_tablet_manager.cc:1367] T df919f52065f444fbc2e9fb59a0494a5 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.007s
I20240502 21:27:06.623011 5941 raft_consensus.cc:348] T df919f52065f444fbc2e9fb59a0494a5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:06.623325 5941 raft_consensus.cc:374] T df919f52065f444fbc2e9fb59a0494a5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:06.623468 5941 raft_consensus.cc:729] T df919f52065f444fbc2e9fb59a0494a5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:27:06.623786 5941 consensus_queue.cc:260] T df919f52065f444fbc2e9fb59a0494a5 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:06.624575 5941 ts_tablet_manager.cc:1398] T df919f52065f444fbc2e9fb59a0494a5 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240502 21:27:06.632802 2032 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.226.133:39863, user_credentials={real_user=slave}} blocked reactor thread for 116058us
I20240502 21:27:06.641703 31388 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=ca5e0c1059364d20a3e8e174f4ed6fa3] alter complete (version 2)
W20240502 21:27:06.674796 2161 connection.cc:535] client connection to 127.24.226.134:43753 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:06.674866 31600 connection.cc:535] client connection to 127.24.226.134:43753 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:06.675089 2161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:06.675122 31600 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:06.675580 31469 connection.cc:535] client connection to 127.24.226.134:43753 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:06.675585 31379 connection.cc:535] client connection to 127.24.226.134:43753 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:06.675801 31379 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:06.675806 31469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:06.676635 31860 connection.cc:535] client connection to 127.24.226.134:43753 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:06.676849 31860 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:06.677454 1767 connection.cc:535] client connection to 127.24.226.134:43753 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:06.677572 2030 connection.cc:535] client connection to 127.24.226.134:43753 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:06.677832 1767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:06.681417 31600 consensus_peers.cc:485] T 925797977edf495ca09aa25988d14fac P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:06.682197 31860 consensus_peers.cc:485] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:06.682548 31730 connection.cc:535] client connection to 127.24.226.134:43753 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:06.682720 31730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:27:06.683642 5960 raft_consensus.cc:482] T 6c84ffd75b4048d886210d6ecbe4d717 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:06.683858 5960 raft_consensus.cc:504] T 6c84ffd75b4048d886210d6ecbe4d717 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
W20240502 21:27:06.685137 31860 consensus_peers.cc:485] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:06.685249 31860 consensus_peers.cc:485] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:06.685967 31730 consensus_peers.cc:485] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:06.686090 31730 consensus_peers.cc:485] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:06.686192 31730 consensus_peers.cc:485] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 08d7a0680d244b55b93d17f9ab57b84c -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:06.686389 5960 leader_election.cc:288] T 6c84ffd75b4048d886210d6ecbe4d717 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:27:06.686633 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c84ffd75b4048d886210d6ecbe4d717" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:27:06.686764 2225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c84ffd75b4048d886210d6ecbe4d717" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:27:06.686842 31925 raft_consensus.cc:2454] T 6c84ffd75b4048d886210d6ecbe4d717 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 0.
I20240502 21:27:06.686959 2225 raft_consensus.cc:2454] T 6c84ffd75b4048d886210d6ecbe4d717 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 0.
I20240502 21:27:06.687294 2033 leader_election.cc:302] T 6c84ffd75b4048d886210d6ecbe4d717 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 2d50f33557944bb480ea32199d9ee4f1; no voters:
W20240502 21:27:06.687667 2161 consensus_peers.cc:485] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:06.687821 2161 consensus_peers.cc:485] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:06.687920 2161 consensus_peers.cc:485] T a8328088928f4de19f7cf20189382d91 P d1406e92aec84ac585304e57bd37a40d -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:06.688035 2161 consensus_peers.cc:485] T 84ba90ce8dd24ea2b325b1dba2e4c494 P d1406e92aec84ac585304e57bd37a40d -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:06.688163 2161 consensus_peers.cc:485] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:06.688342 5960 raft_consensus.cc:2781] T 6c84ffd75b4048d886210d6ecbe4d717 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:06.688829 5960 raft_consensus.cc:482] T 6c84ffd75b4048d886210d6ecbe4d717 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:06.688943 5960 raft_consensus.cc:3037] T 6c84ffd75b4048d886210d6ecbe4d717 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:27:06.689468 31600 consensus_peers.cc:485] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:06.690400 31600 consensus_peers.cc:462] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
W20240502 21:27:06.691255 31600 consensus_peers.cc:485] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:06.692852 1767 leader_election.cc:334] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
W20240502 21:27:06.693316 1767 consensus_peers.cc:485] T b67096e38f2e4c42a33e74f0145a04cc P 1e04157b98dd4fad98858c290b73c3db -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:06.693477 1767 leader_election.cc:334] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
W20240502 21:27:06.693694 31469 consensus_peers.cc:485] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:06.693710 1767 consensus_peers.cc:485] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:06.693769 31469 consensus_peers.cc:485] T e4e50c50be16429faef86d47cc324bdb P a44f41b0b6644fb6aa29476aed8c964f -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:06.693804 31469 consensus_peers.cc:485] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:06.693832 31469 consensus_peers.cc:485] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:06.693817 1767 consensus_peers.cc:485] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:06.693863 31469 consensus_peers.cc:485] T 58be3e32928043e1af387efd99e43b63 P a44f41b0b6644fb6aa29476aed8c964f -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:06.693892 31469 consensus_peers.cc:485] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:06.695459 31379 catalog_manager.cc:4573] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): CreateTablet RPC failed for tablet eed5a5c41d9a46ce8a712b5b399e0472: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
W20240502 21:27:06.678025 2030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:27:06.699102 5960 raft_consensus.cc:504] T 6c84ffd75b4048d886210d6ecbe4d717 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:06.699699 5960 leader_election.cc:288] T 6c84ffd75b4048d886210d6ecbe4d717 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 election: Requested vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:27:06.700251 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c84ffd75b4048d886210d6ecbe4d717" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:27:06.700419 31925 raft_consensus.cc:3037] T 6c84ffd75b4048d886210d6ecbe4d717 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:06.702383 31925 raft_consensus.cc:2454] T 6c84ffd75b4048d886210d6ecbe4d717 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 1.
I20240502 21:27:06.702723 2225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c84ffd75b4048d886210d6ecbe4d717" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:27:06.702920 2225 raft_consensus.cc:3037] T 6c84ffd75b4048d886210d6ecbe4d717 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:06.703014 2033 leader_election.cc:302] T 6c84ffd75b4048d886210d6ecbe4d717 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:27:06.703272 5960 raft_consensus.cc:2781] T 6c84ffd75b4048d886210d6ecbe4d717 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:06.703375 5960 raft_consensus.cc:686] T 6c84ffd75b4048d886210d6ecbe4d717 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEADER]: Becoming Leader. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Running, Role: LEADER
I20240502 21:27:06.703545 5960 consensus_queue.cc:237] T 6c84ffd75b4048d886210d6ecbe4d717 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:06.705420 2225 raft_consensus.cc:2454] T 6c84ffd75b4048d886210d6ecbe4d717 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 1.
I20240502 21:27:06.707702 31388 catalog_manager.cc:5484] T 6c84ffd75b4048d886210d6ecbe4d717 P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135). New cstate: current_term: 1 leader_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:06.720566 5910 raft_consensus.cc:482] T df919f52065f444fbc2e9fb59a0494a5 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:06.720691 5910 raft_consensus.cc:504] T df919f52065f444fbc2e9fb59a0494a5 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:06.721256 5910 leader_election.cc:288] T df919f52065f444fbc2e9fb59a0494a5 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:27:06.721837 2096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df919f52065f444fbc2e9fb59a0494a5" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:27:06.722097 2096 raft_consensus.cc:2454] T df919f52065f444fbc2e9fb59a0494a5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:27:06.722496 31471 leader_election.cc:302] T df919f52065f444fbc2e9fb59a0494a5 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:27:06.722782 5910 raft_consensus.cc:2781] T df919f52065f444fbc2e9fb59a0494a5 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:06.722884 5910 raft_consensus.cc:482] T df919f52065f444fbc2e9fb59a0494a5 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:06.722951 5910 raft_consensus.cc:3037] T df919f52065f444fbc2e9fb59a0494a5 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:27:06.723433 2030 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.226.134:43753, user_credentials={real_user=slave}} blocked reactor thread for 45523.4us
I20240502 21:27:06.725456 5910 raft_consensus.cc:504] T df919f52065f444fbc2e9fb59a0494a5 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:06.726394 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df919f52065f444fbc2e9fb59a0494a5" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:27:06.726554 31925 raft_consensus.cc:2454] T df919f52065f444fbc2e9fb59a0494a5 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:27:06.727393 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df919f52065f444fbc2e9fb59a0494a5" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:27:06.727562 31925 raft_consensus.cc:3037] T df919f52065f444fbc2e9fb59a0494a5 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:06.727499 2096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df919f52065f444fbc2e9fb59a0494a5" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:27:06.727680 2096 raft_consensus.cc:3037] T df919f52065f444fbc2e9fb59a0494a5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:06.730374 2096 raft_consensus.cc:2454] T df919f52065f444fbc2e9fb59a0494a5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:27:06.730804 31471 leader_election.cc:302] T df919f52065f444fbc2e9fb59a0494a5 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:27:06.731012 5705 raft_consensus.cc:2781] T df919f52065f444fbc2e9fb59a0494a5 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:06.731099 5910 leader_election.cc:288] T df919f52065f444fbc2e9fb59a0494a5 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Requested vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:27:06.731141 5705 raft_consensus.cc:686] T df919f52065f444fbc2e9fb59a0494a5 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:27:06.731390 5705 consensus_queue.cc:237] T df919f52065f444fbc2e9fb59a0494a5 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:06.731626 31925 raft_consensus.cc:2454] T df919f52065f444fbc2e9fb59a0494a5 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:27:06.735201 31389 catalog_manager.cc:5484] T df919f52065f444fbc2e9fb59a0494a5 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 0 to 1, leader changed from <none> to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } }
W20240502 21:27:06.739513 2030 consensus_peers.cc:485] T b1d59ebecac34cf7b33424cc73cff030 P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:06.739625 2030 consensus_peers.cc:485] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:06.739737 2030 consensus_peers.cc:485] T 939c34669f6447f9a2a88da769d0f84f P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:06.740341 5417 raft_consensus.cc:1050] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c: attempting to promote NON_VOTER 1ade0354c44c4c37ae480c7a94b11e1b to VOTER
I20240502 21:27:06.741112 5417 consensus_queue.cc:237] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } }
I20240502 21:27:06.749852 31388 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:06.755430 2096 raft_consensus.cc:1261] T 62e70b8e69c94d76b23c655a3f5de5bd P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEARNER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:06.756000 4559 consensus_queue.cc:1035] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:06.763010 31535 raft_consensus.cc:1261] T 62e70b8e69c94d76b23c655a3f5de5bd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:06.765101 4559 consensus_queue.cc:1035] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:06.767700 5934 tablet.cc:1722] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:06.771864 5935 tablet.cc:1722] T 5040da1f38a24860bfd89637cdeab4e7 P 2d50f33557944bb480ea32199d9ee4f1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:06.777724 31665 raft_consensus.cc:1261] T 62e70b8e69c94d76b23c655a3f5de5bd P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:06.779989 5417 consensus_queue.cc:1035] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
W20240502 21:27:06.785789 1767 consensus_peers.cc:485] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:06.786679 31388 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table_new_name [id=ca5e0c1059364d20a3e8e174f4ed6fa3] alter complete (version 3)
I20240502 21:27:06.793962 5990 ts_tablet_manager.cc:902] T 5a54d2aad63c4f3182992f00b8564acf P 1e04157b98dd4fad98858c290b73c3db: Initiating tablet copy from peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:27:06.821695 5990 tablet_copy_client.cc:308] T 5a54d2aad63c4f3182992f00b8564acf P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Beginning tablet copy session from remote peer at address 127.24.226.131:40807
I20240502 21:27:06.822364 31817 tablet_copy_service.cc:140] P 08d7a0680d244b55b93d17f9ab57b84c: Received BeginTabletCopySession request for tablet 5a54d2aad63c4f3182992f00b8564acf from peer 1e04157b98dd4fad98858c290b73c3db ({username='slave'} at 127.24.226.133:41593)
I20240502 21:27:06.822500 31817 tablet_copy_service.cc:161] P 08d7a0680d244b55b93d17f9ab57b84c: Beginning new tablet copy session on tablet 5a54d2aad63c4f3182992f00b8564acf from peer 1e04157b98dd4fad98858c290b73c3db at {username='slave'} at 127.24.226.133:41593: session id = 1e04157b98dd4fad98858c290b73c3db-5a54d2aad63c4f3182992f00b8564acf
I20240502 21:27:06.825491 31817 tablet_copy_source_session.cc:205] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:06.827224 5990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a54d2aad63c4f3182992f00b8564acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:06.842326 5990 tablet_copy_client.cc:783] T 5a54d2aad63c4f3182992f00b8564acf P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:06.842654 5990 tablet_copy_client.cc:647] T 5a54d2aad63c4f3182992f00b8564acf P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:06.853755 5417 raft_consensus.cc:2932] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:27:06.856278 5990 tablet_copy_client.cc:523] T 5a54d2aad63c4f3182992f00b8564acf P 1e04157b98dd4fad98858c290b73c3db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:06.854471 31535 raft_consensus.cc:2932] T 62e70b8e69c94d76b23c655a3f5de5bd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:27:06.861999 5990 tablet_bootstrap.cc:492] T 5a54d2aad63c4f3182992f00b8564acf P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:27:06.871536 31388 catalog_manager.cc:5484] T 62e70b8e69c94d76b23c655a3f5de5bd P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:27:06.890226 31392 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:06.890437 31392 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:06.893464 31392 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 5040da1f38a24860bfd89637cdeab4e7
I20240502 21:27:06.894503 1814 tablet_service.cc:1514] Processing DeleteTablet for tablet 5040da1f38a24860bfd89637cdeab4e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:06 UTC) from {username='slave'} at 127.0.0.1:47510
I20240502 21:27:06.894704 31906 tablet_service.cc:1514] Processing DeleteTablet for tablet 5040da1f38a24860bfd89637cdeab4e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:06 UTC) from {username='slave'} at 127.0.0.1:47824
I20240502 21:27:06.896539 5531 raft_consensus.cc:1050] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f: attempting to promote NON_VOTER 1e04157b98dd4fad98858c290b73c3db to VOTER
I20240502 21:27:06.897294 5531 consensus_queue.cc:237] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } }
I20240502 21:27:06.897570 31666 raft_consensus.cc:2932] T 62e70b8e69c94d76b23c655a3f5de5bd P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:27:06.903211 2093 raft_consensus.cc:2932] T 62e70b8e69c94d76b23c655a3f5de5bd P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { replace: true } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:27:06.922297 6008 tablet_replica.cc:330] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db: stopping tablet replica
I20240502 21:27:06.922535 6008 raft_consensus.cc:2229] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:06.923012 6008 raft_consensus.cc:2258] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:06.923871 6008 ts_tablet_manager.cc:1875] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:06.927202 31797 raft_consensus.cc:1261] T 988a0a9425434cd0b724710e9ebf23a6 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:06.928872 6009 tablet_replica.cc:330] T 5040da1f38a24860bfd89637cdeab4e7 P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:27:06.929162 6009 raft_consensus.cc:2229] T 5040da1f38a24860bfd89637cdeab4e7 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:06.929605 6009 raft_consensus.cc:2258] T 5040da1f38a24860bfd89637cdeab4e7 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:06.929550 6014 consensus_queue.cc:1035] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:06.930532 6009 ts_tablet_manager.cc:1875] T 5040da1f38a24860bfd89637cdeab4e7 P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:06.938511 1834 raft_consensus.cc:1261] T 988a0a9425434cd0b724710e9ebf23a6 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:06.940630 6014 consensus_queue.cc:1035] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:06.948151 5721 raft_consensus.cc:482] T eed5a5c41d9a46ce8a712b5b399e0472 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:06.948251 6009 ts_tablet_manager.cc:1888] T 5040da1f38a24860bfd89637cdeab4e7 P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:06.948356 6009 log.cc:1196] T 5040da1f38a24860bfd89637cdeab4e7 P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/5040da1f38a24860bfd89637cdeab4e7
I20240502 21:27:06.948293 5721 raft_consensus.cc:504] T eed5a5c41d9a46ce8a712b5b399e0472 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:06.949108 5721 leader_election.cc:288] T eed5a5c41d9a46ce8a712b5b399e0472 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:27:06.950979 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eed5a5c41d9a46ce8a712b5b399e0472" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:27:06.951187 31535 raft_consensus.cc:2454] T eed5a5c41d9a46ce8a712b5b399e0472 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:27:06.952555 6014 raft_consensus.cc:482] T eed5a5c41d9a46ce8a712b5b399e0472 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:06.952708 31603 leader_election.cc:302] T eed5a5c41d9a46ce8a712b5b399e0472 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:27:06.952993 6008 ts_tablet_manager.cc:1888] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:06.953269 6008 log.cc:1196] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal/wals/5040da1f38a24860bfd89637cdeab4e7
I20240502 21:27:06.952980 6014 raft_consensus.cc:504] T eed5a5c41d9a46ce8a712b5b399e0472 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:06.954087 6009 ts_tablet_manager.cc:1909] T 5040da1f38a24860bfd89637cdeab4e7 P 2d50f33557944bb480ea32199d9ee4f1: Deleting consensus metadata
I20240502 21:27:06.954335 6014 leader_election.cc:288] T eed5a5c41d9a46ce8a712b5b399e0472 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:27:06.954680 6008 ts_tablet_manager.cc:1909] T 5040da1f38a24860bfd89637cdeab4e7 P 1e04157b98dd4fad98858c290b73c3db: Deleting consensus metadata
W20240502 21:27:06.954967 31600 leader_election.cc:334] T eed5a5c41d9a46ce8a712b5b399e0472 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:06.955443 5721 raft_consensus.cc:2781] T eed5a5c41d9a46ce8a712b5b399e0472 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:06.955552 5721 raft_consensus.cc:482] T eed5a5c41d9a46ce8a712b5b399e0472 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:06.955672 5721 raft_consensus.cc:3037] T eed5a5c41d9a46ce8a712b5b399e0472 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:06.956915 31381 catalog_manager.cc:4830] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): tablet 5040da1f38a24860bfd89637cdeab4e7 (table rebalancer_dynamic_table_new_name [id=ca5e0c1059364d20a3e8e174f4ed6fa3]) successfully deleted
I20240502 21:27:06.957660 1834 raft_consensus.cc:2932] T 988a0a9425434cd0b724710e9ebf23a6 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:27:06.958788 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet 5040da1f38a24860bfd89637cdeab4e7 (table rebalancer_dynamic_table_new_name [id=ca5e0c1059364d20a3e8e174f4ed6fa3]) successfully deleted
W20240502 21:27:06.959002 31469 leader_election.cc:334] T eed5a5c41d9a46ce8a712b5b399e0472 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:06.961403 5721 raft_consensus.cc:504] T eed5a5c41d9a46ce8a712b5b399e0472 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:06.963440 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eed5a5c41d9a46ce8a712b5b399e0472" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:27:06.963585 31535 raft_consensus.cc:3037] T eed5a5c41d9a46ce8a712b5b399e0472 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:06.964560 31392 catalog_manager.cc:5484] T 988a0a9425434cd0b724710e9ebf23a6 P 1e04157b98dd4fad98858c290b73c3db reported cstate change: config changed from index 4 to 5, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:27:06.966017 31535 raft_consensus.cc:2454] T eed5a5c41d9a46ce8a712b5b399e0472 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:27:06.961040 6014 raft_consensus.cc:2932] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:27:06.966513 31603 leader_election.cc:302] T eed5a5c41d9a46ce8a712b5b399e0472 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:27:06.968147 5714 raft_consensus.cc:2781] T eed5a5c41d9a46ce8a712b5b399e0472 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:06.968269 5714 raft_consensus.cc:686] T eed5a5c41d9a46ce8a712b5b399e0472 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Becoming Leader. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Running, Role: LEADER
I20240502 21:27:06.968569 5714 consensus_queue.cc:237] T eed5a5c41d9a46ce8a712b5b399e0472 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:06.971423 31925 raft_consensus.cc:1261] T 988a0a9425434cd0b724710e9ebf23a6 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:06.971408 31797 raft_consensus.cc:2932] T 988a0a9425434cd0b724710e9ebf23a6 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:27:06.974737 5531 consensus_queue.cc:1035] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 21:27:06.976276 31600 leader_election.cc:334] T eed5a5c41d9a46ce8a712b5b399e0472 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:06.980718 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eed5a5c41d9a46ce8a712b5b399e0472" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:27:06.981199 31472 leader_election.cc:302] T eed5a5c41d9a46ce8a712b5b399e0472 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a44f41b0b6644fb6aa29476aed8c964f; no voters: 05f4fac41c4f4959b9b316cb112d0ada, 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:06.985241 5531 raft_consensus.cc:2726] T eed5a5c41d9a46ce8a712b5b399e0472 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:27:06.989264 5990 tablet_bootstrap.cc:492] T 5a54d2aad63c4f3182992f00b8564acf P 1e04157b98dd4fad98858c290b73c3db: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:06.989758 5990 tablet_bootstrap.cc:492] T 5a54d2aad63c4f3182992f00b8564acf P 1e04157b98dd4fad98858c290b73c3db: Bootstrap complete.
I20240502 21:27:06.990084 5990 ts_tablet_manager.cc:1367] T 5a54d2aad63c4f3182992f00b8564acf P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.128s user 0.040s sys 0.010s
I20240502 21:27:06.990988 5990 raft_consensus.cc:348] T 5a54d2aad63c4f3182992f00b8564acf P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } }
I20240502 21:27:06.991328 5990 raft_consensus.cc:729] T 5a54d2aad63c4f3182992f00b8564acf P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: LEARNER
I20240502 21:27:06.991580 5990 consensus_queue.cc:260] T 5a54d2aad63c4f3182992f00b8564acf P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: NON_VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: true } }
I20240502 21:27:06.992509 5990 ts_tablet_manager.cc:1398] T 5a54d2aad63c4f3182992f00b8564acf P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240502 21:27:06.993885 31925 raft_consensus.cc:2932] T 988a0a9425434cd0b724710e9ebf23a6 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:27:06.996169 31392 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:07.005169 2077 tablet_service.cc:1467] Processing CreateTablet for tablet 64c31d44d0224d54a306a29f7a11f568 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0ae9080ead2341c3bac35ab05c4395f1]), partition=
I20240502 21:27:07.005538 2077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64c31d44d0224d54a306a29f7a11f568. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.009675 1814 tablet_service.cc:1467] Processing CreateTablet for tablet 64c31d44d0224d54a306a29f7a11f568 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0ae9080ead2341c3bac35ab05c4395f1]), partition=
I20240502 21:27:07.009994 1814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64c31d44d0224d54a306a29f7a11f568. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.011060 5941 tablet_bootstrap.cc:492] T 64c31d44d0224d54a306a29f7a11f568 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:27:07.012493 5941 tablet_bootstrap.cc:654] T 64c31d44d0224d54a306a29f7a11f568 P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:07.017582 5721 leader_election.cc:288] T eed5a5c41d9a46ce8a712b5b399e0472 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Requested vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:27:07.021915 5941 tablet_bootstrap.cc:492] T 64c31d44d0224d54a306a29f7a11f568 P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:27:07.022046 5941 ts_tablet_manager.cc:1367] T 64c31d44d0224d54a306a29f7a11f568 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.003s
I20240502 21:27:07.022787 5941 raft_consensus.cc:348] T 64c31d44d0224d54a306a29f7a11f568 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:07.023053 5941 raft_consensus.cc:374] T 64c31d44d0224d54a306a29f7a11f568 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:07.023209 5941 raft_consensus.cc:729] T 64c31d44d0224d54a306a29f7a11f568 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:27:07.023461 5941 consensus_queue.cc:260] T 64c31d44d0224d54a306a29f7a11f568 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:07.027629 5941 ts_tablet_manager.cc:1398] T 64c31d44d0224d54a306a29f7a11f568 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 21:27:07.084659 6029 tablet_bootstrap.cc:492] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
W20240502 21:27:07.086496 31472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.226.130:37645, user_credentials={real_user=slave}} blocked reactor thread for 105313us
W20240502 21:27:07.091197 31603 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.226.129:41039, user_credentials={real_user=slave}} blocked reactor thread for 124709us
I20240502 21:27:07.093677 31393 catalog_manager.cc:5484] T eed5a5c41d9a46ce8a712b5b399e0472 P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130). New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:07.096187 6029 tablet_bootstrap.cc:654] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:07.097292 31666 raft_consensus.cc:1203] T 04c07a003bfe472e9142df6082759ae1 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240502 21:27:07.099843 31600 consensus_peers.cc:462] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.104465 6029 tablet_bootstrap.cc:492] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:27:07.104676 6029 ts_tablet_manager.cc:1367] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20240502 21:27:07.105495 6029 raft_consensus.cc:348] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:07.105767 6029 raft_consensus.cc:374] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:07.105903 6029 raft_consensus.cc:729] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:27:07.106209 6029 consensus_queue.cc:260] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:07.107846 6029 ts_tablet_manager.cc:1398] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:27:07.112825 5661 raft_consensus.cc:482] T 51551a1967b74094b373cc030e2bf850 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.113013 5661 raft_consensus.cc:504] T 51551a1967b74094b373cc030e2bf850 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } }
I20240502 21:27:07.113865 5661 leader_election.cc:288] T 51551a1967b74094b373cc030e2bf850 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
W20240502 21:27:07.117404 2161 leader_election.cc:334] T 51551a1967b74094b373cc030e2bf850 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.117928 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51551a1967b74094b373cc030e2bf850" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:27:07.118718 2164 leader_election.cc:302] T 51551a1967b74094b373cc030e2bf850 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1406e92aec84ac585304e57bd37a40d; no voters: 9dec645b473f40578e726f3e50e219bf, a44f41b0b6644fb6aa29476aed8c964f
I20240502 21:27:07.119459 5522 raft_consensus.cc:2726] T 51551a1967b74094b373cc030e2bf850 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:27:07.126473 31817 tablet_copy_service.cc:342] P 08d7a0680d244b55b93d17f9ab57b84c: Request end of tablet copy session 1e04157b98dd4fad98858c290b73c3db-5a54d2aad63c4f3182992f00b8564acf received from {username='slave'} at 127.24.226.133:41593
I20240502 21:27:07.126600 31817 tablet_copy_service.cc:433] P 08d7a0680d244b55b93d17f9ab57b84c: ending tablet copy session 1e04157b98dd4fad98858c290b73c3db-5a54d2aad63c4f3182992f00b8564acf on tablet 5a54d2aad63c4f3182992f00b8564acf with peer 1e04157b98dd4fad98858c290b73c3db
I20240502 21:27:07.128583 5910 consensus_queue.cc:1035] T df919f52065f444fbc2e9fb59a0494a5 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:07.129314 31797 consensus_queue.cc:237] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } }
I20240502 21:27:07.131650 31535 raft_consensus.cc:1261] T 62e70b8e69c94d76b23c655a3f5de5bd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:07.131687 2093 raft_consensus.cc:1261] T 62e70b8e69c94d76b23c655a3f5de5bd P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:07.131979 31775 tablet_service.cc:1467] Processing CreateTablet for tablet 64c31d44d0224d54a306a29f7a11f568 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0ae9080ead2341c3bac35ab05c4395f1]), partition=
I20240502 21:27:07.132166 6000 consensus_queue.cc:1035] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:07.132362 31775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64c31d44d0224d54a306a29f7a11f568. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.132606 5417 consensus_queue.cc:1035] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:07.138875 6004 raft_consensus.cc:2932] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:27:07.140007 5954 consensus_queue.cc:1035] T 6c84ffd75b4048d886210d6ecbe4d717 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:07.141677 31535 raft_consensus.cc:2932] T 62e70b8e69c94d76b23c655a3f5de5bd P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:27:07.145085 2093 raft_consensus.cc:2932] T 62e70b8e69c94d76b23c655a3f5de5bd P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:27:07.146458 5705 raft_consensus.cc:482] T acb3bdc518b84b5abc362cc4e12027c3 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.146612 5705 raft_consensus.cc:504] T acb3bdc518b84b5abc362cc4e12027c3 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } }
I20240502 21:27:07.147361 5705 leader_election.cc:288] T acb3bdc518b84b5abc362cc4e12027c3 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:27:07.150139 1834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3bdc518b84b5abc362cc4e12027c3" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
W20240502 21:27:07.151098 31469 leader_election.cc:334] T acb3bdc518b84b5abc362cc4e12027c3 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.151805 31471 leader_election.cc:302] T acb3bdc518b84b5abc362cc4e12027c3 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a44f41b0b6644fb6aa29476aed8c964f; no voters: 1e04157b98dd4fad98858c290b73c3db, 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:07.152626 31388 catalog_manager.cc:5484] T 62e70b8e69c94d76b23c655a3f5de5bd P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: config changed from index 5 to 6, VOTER 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) evicted. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { promote: false } } }
I20240502 21:27:07.154846 6016 raft_consensus.cc:2726] T acb3bdc518b84b5abc362cc4e12027c3 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:27:07.156921 31645 tablet_service.cc:1514] Processing DeleteTablet for tablet 62e70b8e69c94d76b23c655a3f5de5bd with delete_type TABLET_DATA_TOMBSTONED (TS 05f4fac41c4f4959b9b316cb112d0ada not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52130
I20240502 21:27:07.158826 6033 tablet_replica.cc:330] T 62e70b8e69c94d76b23c655a3f5de5bd P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:27:07.159107 6033 raft_consensus.cc:2229] T 62e70b8e69c94d76b23c655a3f5de5bd P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:07.159379 6033 raft_consensus.cc:2258] T 62e70b8e69c94d76b23c655a3f5de5bd P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:07.161409 6033 ts_tablet_manager.cc:1875] T 62e70b8e69c94d76b23c655a3f5de5bd P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:07.166685 6036 tablet_bootstrap.cc:492] T 64c31d44d0224d54a306a29f7a11f568 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:27:07.168332 6036 tablet_bootstrap.cc:654] T 64c31d44d0224d54a306a29f7a11f568 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:07.171360 6033 ts_tablet_manager.cc:1888] T 62e70b8e69c94d76b23c655a3f5de5bd P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:07.171861 5215 raft_consensus.cc:482] T 69432f98b88740adb4fcbc112298cf20 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.172039 5215 raft_consensus.cc:504] T 69432f98b88740adb4fcbc112298cf20 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:07.172751 6033 log.cc:1196] T 62e70b8e69c94d76b23c655a3f5de5bd P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/62e70b8e69c94d76b23c655a3f5de5bd
I20240502 21:27:07.172744 5215 leader_election.cc:288] T 69432f98b88740adb4fcbc112298cf20 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:27:07.173936 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet 62e70b8e69c94d76b23c655a3f5de5bd (table rebalance_test_table_1 [id=08ebfd17dee84efd8a264a88ead3fa46]) successfully deleted
I20240502 21:27:07.174358 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69432f98b88740adb4fcbc112298cf20" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
W20240502 21:27:07.174861 1767 leader_election.cc:334] T 69432f98b88740adb4fcbc112298cf20 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.175007 1767 leader_election.cc:302] T 69432f98b88740adb4fcbc112298cf20 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db; no voters: 05f4fac41c4f4959b9b316cb112d0ada, 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:07.175277 5854 raft_consensus.cc:2726] T 69432f98b88740adb4fcbc112298cf20 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:27:07.176935 6036 tablet_bootstrap.cc:492] T 64c31d44d0224d54a306a29f7a11f568 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:27:07.177111 6036 ts_tablet_manager.cc:1367] T 64c31d44d0224d54a306a29f7a11f568 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240502 21:27:07.177335 31381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 62e70b8e69c94d76b23c655a3f5de5bd with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:07.177762 6036 raft_consensus.cc:348] T 64c31d44d0224d54a306a29f7a11f568 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:07.177922 6036 raft_consensus.cc:374] T 64c31d44d0224d54a306a29f7a11f568 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:07.177985 6036 raft_consensus.cc:729] T 64c31d44d0224d54a306a29f7a11f568 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:27:07.179940 6036 consensus_queue.cc:260] T 64c31d44d0224d54a306a29f7a11f568 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:07.181205 6036 ts_tablet_manager.cc:1398] T 64c31d44d0224d54a306a29f7a11f568 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:07.181555 1834 raft_consensus.cc:1203] T 5a54d2aad63c4f3182992f00b8564acf P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:27:07.193102 5910 consensus_queue.cc:1035] T df919f52065f444fbc2e9fb59a0494a5 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:07.207558 5960 consensus_queue.cc:1035] T 6c84ffd75b4048d886210d6ecbe4d717 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:07.235882 5854 raft_consensus.cc:482] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:07.236093 5854 raft_consensus.cc:504] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:07.236896 5854 leader_election.cc:288] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:27:07.238269 2093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c31d44d0224d54a306a29f7a11f568" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:27:07.238534 2093 raft_consensus.cc:2454] T 64c31d44d0224d54a306a29f7a11f568 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 0.
I20240502 21:27:07.240268 1769 leader_election.cc:302] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 1e04157b98dd4fad98858c290b73c3db; no voters:
I20240502 21:27:07.241891 6024 raft_consensus.cc:482] T f02f83ec0a494da5b66241d9b379a03f P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.242045 6024 raft_consensus.cc:504] T f02f83ec0a494da5b66241d9b379a03f P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } }
I20240502 21:27:07.242791 6024 leader_election.cc:288] T f02f83ec0a494da5b66241d9b379a03f P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
W20240502 21:27:07.246119 31469 leader_election.cc:334] T f02f83ec0a494da5b66241d9b379a03f P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.246948 5215 raft_consensus.cc:2781] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:07.247071 5215 raft_consensus.cc:482] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:07.247216 5215 raft_consensus.cc:3037] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:07.250447 5960 raft_consensus.cc:482] T 41029f1332c64ba58ee0ee45714ba2a8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.250581 5960 raft_consensus.cc:504] T 41029f1332c64ba58ee0ee45714ba2a8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:07.250864 5215 raft_consensus.cc:504] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:07.251152 5960 leader_election.cc:288] T 41029f1332c64ba58ee0ee45714ba2a8 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:27:07.251808 31924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41029f1332c64ba58ee0ee45714ba2a8" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
W20240502 21:27:07.252756 2030 leader_election.cc:334] T 41029f1332c64ba58ee0ee45714ba2a8 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.252893 2030 leader_election.cc:302] T 41029f1332c64ba58ee0ee45714ba2a8 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b; no voters: 2d50f33557944bb480ea32199d9ee4f1, 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:07.253278 5960 raft_consensus.cc:2726] T 41029f1332c64ba58ee0ee45714ba2a8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:27:07.254004 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c31d44d0224d54a306a29f7a11f568" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:27:07.254213 31797 raft_consensus.cc:2454] T 64c31d44d0224d54a306a29f7a11f568 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 0.
I20240502 21:27:07.254524 31796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c31d44d0224d54a306a29f7a11f568" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:27:07.254678 31796 raft_consensus.cc:3037] T 64c31d44d0224d54a306a29f7a11f568 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:07.257653 31796 raft_consensus.cc:2454] T 64c31d44d0224d54a306a29f7a11f568 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:27:07.258064 1769 leader_election.cc:302] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 1e04157b98dd4fad98858c290b73c3db; no voters:
I20240502 21:27:07.258355 5854 raft_consensus.cc:2781] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:07.258482 5854 raft_consensus.cc:686] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Becoming Leader. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Running, Role: LEADER
I20240502 21:27:07.258739 5854 consensus_queue.cc:237] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:07.262843 2093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c31d44d0224d54a306a29f7a11f568" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:27:07.263036 2093 raft_consensus.cc:3037] T 64c31d44d0224d54a306a29f7a11f568 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:07.267015 5896 raft_consensus.cc:482] T 41029f1332c64ba58ee0ee45714ba2a8 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.267256 5896 raft_consensus.cc:504] T 41029f1332c64ba58ee0ee45714ba2a8 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:07.268026 5896 leader_election.cc:288] T 41029f1332c64ba58ee0ee45714ba2a8 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:27:07.268241 5896 raft_consensus.cc:482] T b407c2c9b3a14ce3ada52b7856c6ecee P 2d50f33557944bb480ea32199d9ee4f1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.269557 2095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41029f1332c64ba58ee0ee45714ba2a8" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:27:07.269785 2095 raft_consensus.cc:2454] T 41029f1332c64ba58ee0ee45714ba2a8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:27:07.266446 2093 raft_consensus.cc:2454] T 64c31d44d0224d54a306a29f7a11f568 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
W20240502 21:27:07.270241 31860 leader_election.cc:334] T 41029f1332c64ba58ee0ee45714ba2a8 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.270782 31862 leader_election.cc:302] T 41029f1332c64ba58ee0ee45714ba2a8 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 2d50f33557944bb480ea32199d9ee4f1; no voters: 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:07.271420 5512 raft_consensus.cc:2781] T 41029f1332c64ba58ee0ee45714ba2a8 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:27:07.271559 5512 raft_consensus.cc:482] T 41029f1332c64ba58ee0ee45714ba2a8 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.271661 5512 raft_consensus.cc:3037] T 41029f1332c64ba58ee0ee45714ba2a8 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:07.275228 5512 raft_consensus.cc:504] T 41029f1332c64ba58ee0ee45714ba2a8 P 2d50f33557944bb480ea32199d9ee4f1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
W20240502 21:27:07.277240 31860 leader_election.cc:334] T 41029f1332c64ba58ee0ee45714ba2a8 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.278277 31393 catalog_manager.cc:5484] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e04157b98dd4fad98858c290b73c3db (127.24.226.133). New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:07.280149 5896 raft_consensus.cc:504] T b407c2c9b3a14ce3ada52b7856c6ecee P 2d50f33557944bb480ea32199d9ee4f1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:27:07.281042 5896 leader_election.cc:288] T b407c2c9b3a14ce3ada52b7856c6ecee P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:27:07.281424 5512 leader_election.cc:288] T 41029f1332c64ba58ee0ee45714ba2a8 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 2 election: Requested vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:27:07.281710 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b407c2c9b3a14ce3ada52b7856c6ecee" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:27:07.282706 2093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41029f1332c64ba58ee0ee45714ba2a8" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:27:07.283957 2093 raft_consensus.cc:3037] T 41029f1332c64ba58ee0ee45714ba2a8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:07.284876 5215 leader_election.cc:288] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 1 election: Requested vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
W20240502 21:27:07.285231 31860 leader_election.cc:334] T b407c2c9b3a14ce3ada52b7856c6ecee P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.285355 31860 leader_election.cc:302] T b407c2c9b3a14ce3ada52b7856c6ecee P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1; no voters: 05f4fac41c4f4959b9b316cb112d0ada, 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:07.288565 5896 raft_consensus.cc:2726] T b407c2c9b3a14ce3ada52b7856c6ecee P 2d50f33557944bb480ea32199d9ee4f1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 21:27:07.292662 2096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02f83ec0a494da5b66241d9b379a03f" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:27:07.293102 31471 leader_election.cc:302] T f02f83ec0a494da5b66241d9b379a03f P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a44f41b0b6644fb6aa29476aed8c964f; no voters: 1ade0354c44c4c37ae480c7a94b11e1b, 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:07.293344 5705 raft_consensus.cc:2726] T f02f83ec0a494da5b66241d9b379a03f P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:27:07.296965 2093 raft_consensus.cc:2454] T 41029f1332c64ba58ee0ee45714ba2a8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 2.
I20240502 21:27:07.300330 31862 leader_election.cc:302] T 41029f1332c64ba58ee0ee45714ba2a8 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 2d50f33557944bb480ea32199d9ee4f1; no voters: 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:07.300992 5896 raft_consensus.cc:2781] T 41029f1332c64ba58ee0ee45714ba2a8 P 2d50f33557944bb480ea32199d9ee4f1 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:07.301118 5896 raft_consensus.cc:686] T 41029f1332c64ba58ee0ee45714ba2a8 P 2d50f33557944bb480ea32199d9ee4f1 [term 2 LEADER]: Becoming Leader. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Running, Role: LEADER
I20240502 21:27:07.301357 5896 consensus_queue.cc:237] T 41029f1332c64ba58ee0ee45714ba2a8 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:07.302039 6027 raft_consensus.cc:482] T f9bbc56ba2f841b9bb36a864e1a1f808 P 05f4fac41c4f4959b9b316cb112d0ada [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.302225 6027 raft_consensus.cc:504] T f9bbc56ba2f841b9bb36a864e1a1f808 P 05f4fac41c4f4959b9b316cb112d0ada [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:27:07.302794 6027 leader_election.cc:288] T f9bbc56ba2f841b9bb36a864e1a1f808 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:27:07.303347 31924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9bbc56ba2f841b9bb36a864e1a1f808" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
W20240502 21:27:07.304339 31600 leader_election.cc:334] T f9bbc56ba2f841b9bb36a864e1a1f808 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.304445 31600 leader_election.cc:302] T f9bbc56ba2f841b9bb36a864e1a1f808 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada; no voters: 2d50f33557944bb480ea32199d9ee4f1, 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:07.304961 6027 raft_consensus.cc:2726] T f9bbc56ba2f841b9bb36a864e1a1f808 P 05f4fac41c4f4959b9b316cb112d0ada [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 21:27:07.308784 5910 raft_consensus.cc:482] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.308954 5910 raft_consensus.cc:504] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } }
I20240502 21:27:07.309666 5910 leader_election.cc:288] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:27:07.310763 31392 catalog_manager.cc:5484] T 41029f1332c64ba58ee0ee45714ba2a8 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: term changed from 1 to 2, leader changed from 9dec645b473f40578e726f3e50e219bf (127.24.226.134) to 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132). New cstate: current_term: 2 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } }
W20240502 21:27:07.311192 31469 leader_election.cc:334] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.311357 2225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51551a1967b74094b373cc030e2bf850" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:27:07.311575 2225 raft_consensus.cc:2454] T 51551a1967b74094b373cc030e2bf850 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:27:07.311879 31471 leader_election.cc:302] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a44f41b0b6644fb6aa29476aed8c964f, d1406e92aec84ac585304e57bd37a40d; no voters: 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:07.312233 5705 raft_consensus.cc:2781] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:27:07.312296 5705 raft_consensus.cc:482] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.312353 5705 raft_consensus.cc:3037] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:07.314749 5705 raft_consensus.cc:504] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } }
I20240502 21:27:07.315248 5705 leader_election.cc:288] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 2 election: Requested vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
W20240502 21:27:07.316567 31469 leader_election.cc:334] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.317221 2225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51551a1967b74094b373cc030e2bf850" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:27:07.317366 5896 raft_consensus.cc:482] T c977ca41b72d4d45ac7146fbca350824 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.317412 2225 raft_consensus.cc:3037] T 51551a1967b74094b373cc030e2bf850 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:07.317540 5896 raft_consensus.cc:504] T c977ca41b72d4d45ac7146fbca350824 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:07.318228 5896 leader_election.cc:288] T c977ca41b72d4d45ac7146fbca350824 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:27:07.319070 1834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c977ca41b72d4d45ac7146fbca350824" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:27:07.319659 2225 raft_consensus.cc:2454] T 51551a1967b74094b373cc030e2bf850 P d1406e92aec84ac585304e57bd37a40d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 2.
I20240502 21:27:07.320001 31471 leader_election.cc:302] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a44f41b0b6644fb6aa29476aed8c964f, d1406e92aec84ac585304e57bd37a40d; no voters: 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:07.320611 5910 raft_consensus.cc:2781] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:07.320724 5910 raft_consensus.cc:686] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f [term 2 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:27:07.320941 5910 consensus_queue.cc:237] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } }
W20240502 21:27:07.323714 31860 leader_election.cc:334] T c977ca41b72d4d45ac7146fbca350824 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.323860 31860 leader_election.cc:302] T c977ca41b72d4d45ac7146fbca350824 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1; no voters: 1e04157b98dd4fad98858c290b73c3db, 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:07.324294 5896 raft_consensus.cc:2726] T c977ca41b72d4d45ac7146fbca350824 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:27:07.327558 31392 catalog_manager.cc:5484] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 1 to 2, leader changed from 9dec645b473f40578e726f3e50e219bf (127.24.226.134) to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 2 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:07.337487 5910 raft_consensus.cc:482] T b20c2bb60f6149458a62ee443b8cf3ab P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.337600 5910 raft_consensus.cc:504] T b20c2bb60f6149458a62ee443b8cf3ab P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } }
I20240502 21:27:07.338104 5910 leader_election.cc:288] T b20c2bb60f6149458a62ee443b8cf3ab P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:27:07.340337 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b20c2bb60f6149458a62ee443b8cf3ab" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
W20240502 21:27:07.343544 31469 leader_election.cc:334] T b20c2bb60f6149458a62ee443b8cf3ab P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.343640 31469 leader_election.cc:302] T b20c2bb60f6149458a62ee443b8cf3ab P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a44f41b0b6644fb6aa29476aed8c964f; no voters: 05f4fac41c4f4959b9b316cb112d0ada, 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:07.344077 5910 raft_consensus.cc:2726] T b20c2bb60f6149458a62ee443b8cf3ab P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:27:07.357993 5896 raft_consensus.cc:482] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.358223 5896 raft_consensus.cc:504] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:27:07.358966 5896 leader_election.cc:288] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:27:07.360424 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9bbc56ba2f841b9bb36a864e1a1f808" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:27:07.360678 31666 raft_consensus.cc:2454] T f9bbc56ba2f841b9bb36a864e1a1f808 P 05f4fac41c4f4959b9b316cb112d0ada [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 2.
I20240502 21:27:07.361316 31863 leader_election.cc:302] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 2d50f33557944bb480ea32199d9ee4f1; no voters:
W20240502 21:27:07.398610 31860 leader_election.cc:334] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.402328 5896 raft_consensus.cc:2781] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 21:27:07.402469 5896 raft_consensus.cc:482] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.402572 5896 raft_consensus.cc:3037] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:07.406095 5896 raft_consensus.cc:504] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:27:07.406930 5896 leader_election.cc:288] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 3 election: Requested vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
W20240502 21:27:07.408344 31863 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.226.130:37645, user_credentials={real_user=slave}} blocked reactor thread for 47054.3us
W20240502 21:27:07.408515 31860 leader_election.cc:334] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.409219 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9bbc56ba2f841b9bb36a864e1a1f808" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:27:07.409446 31666 raft_consensus.cc:3037] T f9bbc56ba2f841b9bb36a864e1a1f808 P 05f4fac41c4f4959b9b316cb112d0ada [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:07.411695 31666 raft_consensus.cc:2454] T f9bbc56ba2f841b9bb36a864e1a1f808 P 05f4fac41c4f4959b9b316cb112d0ada [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 3.
I20240502 21:27:07.412331 31863 leader_election.cc:302] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 2d50f33557944bb480ea32199d9ee4f1; no voters: 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:07.412668 5896 raft_consensus.cc:2781] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:07.412832 5896 raft_consensus.cc:686] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 [term 3 LEADER]: Becoming Leader. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Running, Role: LEADER
I20240502 21:27:07.413123 5896 consensus_queue.cc:237] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:27:07.422155 31389 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:07.423012 5643 raft_consensus.cc:1050] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db: attempting to promote NON_VOTER 05f4fac41c4f4959b9b316cb112d0ada to VOTER
I20240502 21:27:07.424106 5643 consensus_queue.cc:237] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } }
I20240502 21:27:07.438838 5661 raft_consensus.cc:482] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.439072 5661 raft_consensus.cc:504] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:27:07.439898 5661 leader_election.cc:288] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:27:07.440636 2093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed9c8356c4c4a5e94b755d58b3eb7a0" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
W20240502 21:27:07.449051 2161 leader_election.cc:334] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.449241 2161 leader_election.cc:302] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1406e92aec84ac585304e57bd37a40d; no voters: 1ade0354c44c4c37ae480c7a94b11e1b, 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:07.450201 5661 raft_consensus.cc:2726] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:27:07.451020 5661 raft_consensus.cc:482] T 565f30d5539c4b7ea956dc49d0f9a55e P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.451220 5661 raft_consensus.cc:504] T 565f30d5539c4b7ea956dc49d0f9a55e P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:07.451964 5661 leader_election.cc:288] T 565f30d5539c4b7ea956dc49d0f9a55e P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:27:07.458012 2096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "565f30d5539c4b7ea956dc49d0f9a55e" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
W20240502 21:27:07.463021 2161 leader_election.cc:334] T 565f30d5539c4b7ea956dc49d0f9a55e P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.463156 2161 leader_election.cc:302] T 565f30d5539c4b7ea956dc49d0f9a55e P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1406e92aec84ac585304e57bd37a40d; no voters: 1ade0354c44c4c37ae480c7a94b11e1b, 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:07.463657 5661 raft_consensus.cc:2726] T 565f30d5539c4b7ea956dc49d0f9a55e P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:27:07.472909 31535 consensus_queue.cc:237] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } }
I20240502 21:27:07.475107 5714 raft_consensus.cc:482] T 69432f98b88740adb4fcbc112298cf20 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.475275 5714 raft_consensus.cc:504] T 69432f98b88740adb4fcbc112298cf20 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:07.475926 5714 leader_election.cc:288] T 69432f98b88740adb4fcbc112298cf20 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:27:07.476511 1834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69432f98b88740adb4fcbc112298cf20" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:27:07.476720 1834 raft_consensus.cc:2454] T 69432f98b88740adb4fcbc112298cf20 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:27:07.476927 31924 raft_consensus.cc:1261] T 988a0a9425434cd0b724710e9ebf23a6 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:07.477141 31602 leader_election.cc:302] T 69432f98b88740adb4fcbc112298cf20 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 1e04157b98dd4fad98858c290b73c3db; no voters:
I20240502 21:27:07.478122 5705 consensus_queue.cc:1035] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:07.480377 6027 raft_consensus.cc:2781] T 69432f98b88740adb4fcbc112298cf20 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:27:07.480480 6027 raft_consensus.cc:482] T 69432f98b88740adb4fcbc112298cf20 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.480558 6027 raft_consensus.cc:3037] T 69432f98b88740adb4fcbc112298cf20 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Advancing to term 2
W20240502 21:27:07.483964 31600 leader_election.cc:334] T 69432f98b88740adb4fcbc112298cf20 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.483206 6027 raft_consensus.cc:504] T 69432f98b88740adb4fcbc112298cf20 P 05f4fac41c4f4959b9b316cb112d0ada [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:07.491241 1834 raft_consensus.cc:1261] T 988a0a9425434cd0b724710e9ebf23a6 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:07.492520 5705 raft_consensus.cc:2932] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:27:07.497431 5910 consensus_queue.cc:1035] T 988a0a9425434cd0b724710e9ebf23a6 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:07.502077 31924 raft_consensus.cc:2932] T 988a0a9425434cd0b724710e9ebf23a6 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:27:07.507357 1834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69432f98b88740adb4fcbc112298cf20" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:27:07.507602 1834 raft_consensus.cc:3037] T 69432f98b88740adb4fcbc112298cf20 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:07.508477 6027 leader_election.cc:288] T 69432f98b88740adb4fcbc112298cf20 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 2 election: Requested vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:27:07.511211 1831 raft_consensus.cc:2932] T 988a0a9425434cd0b724710e9ebf23a6 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
W20240502 21:27:07.522716 31600 leader_election.cc:334] T 69432f98b88740adb4fcbc112298cf20 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
W20240502 21:27:07.526185 31600 consensus_peers.cc:462] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.529063 1834 raft_consensus.cc:2454] T 69432f98b88740adb4fcbc112298cf20 P 1e04157b98dd4fad98858c290b73c3db [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 2.
I20240502 21:27:07.537321 31394 catalog_manager.cc:5484] T 988a0a9425434cd0b724710e9ebf23a6 P 1e04157b98dd4fad98858c290b73c3db reported cstate change: config changed from index 5 to 6, VOTER 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131) evicted. New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:27:07.543015 5960 raft_consensus.cc:482] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.543180 5960 raft_consensus.cc:504] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:27:07.543800 5960 leader_election.cc:288] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:27:07.544346 2224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed9c8356c4c4a5e94b755d58b3eb7a0" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:27:07.544572 2224 raft_consensus.cc:2454] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 1.
I20240502 21:27:07.545095 2032 leader_election.cc:302] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:27:07.545615 5960 raft_consensus.cc:2781] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:27:07.545648 31775 tablet_service.cc:1514] Processing DeleteTablet for tablet 988a0a9425434cd0b724710e9ebf23a6 with delete_type TABLET_DATA_TOMBSTONED (TS 08d7a0680d244b55b93d17f9ab57b84c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51596
I20240502 21:27:07.545734 5960 raft_consensus.cc:482] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.545836 5960 raft_consensus.cc:3037] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Advancing to term 2
W20240502 21:27:07.547395 2030 leader_election.cc:334] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.551362 5960 raft_consensus.cc:504] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 1ade0354c44c4c37ae480c7a94b11e1b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:27:07.552237 2224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed9c8356c4c4a5e94b755d58b3eb7a0" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:27:07.552402 2224 raft_consensus.cc:3037] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:07.554826 2224 raft_consensus.cc:2454] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P d1406e92aec84ac585304e57bd37a40d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 2.
I20240502 21:27:07.555318 2032 leader_election.cc:302] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:27:07.555622 5945 raft_consensus.cc:2781] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 1ade0354c44c4c37ae480c7a94b11e1b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:07.555758 5945 raft_consensus.cc:686] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 1ade0354c44c4c37ae480c7a94b11e1b [term 2 LEADER]: Becoming Leader. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Running, Role: LEADER
I20240502 21:27:07.555930 6027 raft_consensus.cc:482] T 78f06bc68b954dc4b55b3c8c3c612c2f P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.556056 5945 consensus_queue.cc:237] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:27:07.557296 5215 raft_consensus.cc:482] T a8684b1ca3a642688a232afaf1b28f96 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.557595 5215 raft_consensus.cc:504] T a8684b1ca3a642688a232afaf1b28f96 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:27:07.558225 5215 leader_election.cc:288] T a8684b1ca3a642688a232afaf1b28f96 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
W20240502 21:27:07.558861 2030 leader_election.cc:334] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.559056 5960 leader_election.cc:288] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 election: Requested vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
W20240502 21:27:07.560375 1767 leader_election.cc:334] T a8684b1ca3a642688a232afaf1b28f96 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.561839 5215 raft_consensus.cc:482] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.561970 5215 raft_consensus.cc:504] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } }
I20240502 21:27:07.562670 5215 leader_election.cc:288] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
W20240502 21:27:07.565986 1767 leader_election.cc:334] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.566313 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13c3b9c4b7c471fa34156b781473ec4" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:27:07.566947 1770 leader_election.cc:302] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db; no voters: 9dec645b473f40578e726f3e50e219bf, a44f41b0b6644fb6aa29476aed8c964f
I20240502 21:27:07.567303 5215 raft_consensus.cc:2726] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 21:27:07.556067 6027 raft_consensus.cc:504] T 78f06bc68b954dc4b55b3c8c3c612c2f P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:07.568214 31602 leader_election.cc:302] T 69432f98b88740adb4fcbc112298cf20 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 1e04157b98dd4fad98858c290b73c3db; no voters: 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:07.568696 6021 raft_consensus.cc:2781] T 69432f98b88740adb4fcbc112298cf20 P 05f4fac41c4f4959b9b316cb112d0ada [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:07.568814 6021 raft_consensus.cc:686] T 69432f98b88740adb4fcbc112298cf20 P 05f4fac41c4f4959b9b316cb112d0ada [term 2 LEADER]: Becoming Leader. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Running, Role: LEADER
I20240502 21:27:07.569026 2224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8684b1ca3a642688a232afaf1b28f96" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:27:07.569096 6021 consensus_queue.cc:237] T 69432f98b88740adb4fcbc112298cf20 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:07.569525 1769 leader_election.cc:302] T a8684b1ca3a642688a232afaf1b28f96 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db; no voters: 9dec645b473f40578e726f3e50e219bf, d1406e92aec84ac585304e57bd37a40d
I20240502 21:27:07.570775 5215 raft_consensus.cc:2726] T a8684b1ca3a642688a232afaf1b28f96 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:27:07.574049 6027 leader_election.cc:288] T 78f06bc68b954dc4b55b3c8c3c612c2f P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:27:07.574939 5954 raft_consensus.cc:482] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.575142 5954 raft_consensus.cc:504] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } }
W20240502 21:27:07.575383 31600 leader_election.cc:334] T 78f06bc68b954dc4b55b3c8c3c612c2f P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
W20240502 21:27:07.575541 31600 consensus_peers.cc:485] T eed5a5c41d9a46ce8a712b5b399e0472 P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:07.575925 5954 leader_election.cc:288] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:27:07.577324 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02f83ec0a494da5b66241d9b379a03f" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:27:07.577503 31535 raft_consensus.cc:2454] T f02f83ec0a494da5b66241d9b379a03f P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 1.
I20240502 21:27:07.578042 2033 leader_election.cc:302] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, a44f41b0b6644fb6aa29476aed8c964f; no voters:
W20240502 21:27:07.580150 2030 leader_election.cc:334] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.581056 6004 raft_consensus.cc:1050] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c: attempting to promote NON_VOTER 1e04157b98dd4fad98858c290b73c3db to VOTER
I20240502 21:27:07.581786 6004 consensus_queue.cc:237] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } }
I20240502 21:27:07.583124 5954 raft_consensus.cc:2781] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:27:07.583233 5954 raft_consensus.cc:482] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.583341 5954 raft_consensus.cc:3037] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:07.585851 5661 raft_consensus.cc:482] T a8684b1ca3a642688a232afaf1b28f96 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.586011 5661 raft_consensus.cc:504] T a8684b1ca3a642688a232afaf1b28f96 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:27:07.586771 5661 leader_election.cc:288] T a8684b1ca3a642688a232afaf1b28f96 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:27:07.587198 31382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 988a0a9425434cd0b724710e9ebf23a6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:07.587182 1831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8684b1ca3a642688a232afaf1b28f96" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:27:07.587389 1831 raft_consensus.cc:2454] T a8684b1ca3a642688a232afaf1b28f96 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 1.
I20240502 21:27:07.587674 6024 raft_consensus.cc:482] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.587738 2163 leader_election.cc:302] T a8684b1ca3a642688a232afaf1b28f96 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:27:07.587800 6024 raft_consensus.cc:504] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } }
I20240502 21:27:07.588042 5661 raft_consensus.cc:2781] T a8684b1ca3a642688a232afaf1b28f96 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:27:07.588122 5661 raft_consensus.cc:482] T a8684b1ca3a642688a232afaf1b28f96 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.588213 5661 raft_consensus.cc:3037] T a8684b1ca3a642688a232afaf1b28f96 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:07.588413 6024 leader_election.cc:288] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:27:07.588533 6024 raft_consensus.cc:482] T 78f06bc68b954dc4b55b3c8c3c612c2f P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.588624 6024 raft_consensus.cc:504] T 78f06bc68b954dc4b55b3c8c3c612c2f P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:07.589139 6024 leader_election.cc:288] T 78f06bc68b954dc4b55b3c8c3c612c2f P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:27:07.590850 5661 raft_consensus.cc:504] T a8684b1ca3a642688a232afaf1b28f96 P d1406e92aec84ac585304e57bd37a40d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
W20240502 21:27:07.591118 31469 leader_election.cc:334] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
W20240502 21:27:07.591226 31469 leader_election.cc:334] T 78f06bc68b954dc4b55b3c8c3c612c2f P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.591742 5661 leader_election.cc:288] T a8684b1ca3a642688a232afaf1b28f96 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 election: Requested vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:27:07.592042 6070 tablet_replica.cc:330] T 988a0a9425434cd0b724710e9ebf23a6 P 08d7a0680d244b55b93d17f9ab57b84c: stopping tablet replica
I20240502 21:27:07.592038 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f06bc68b954dc4b55b3c8c3c612c2f" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:27:07.592198 31666 raft_consensus.cc:2454] T 78f06bc68b954dc4b55b3c8c3c612c2f P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:27:07.592218 6070 raft_consensus.cc:2229] T 988a0a9425434cd0b724710e9ebf23a6 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:07.592386 6070 raft_consensus.cc:2258] T 988a0a9425434cd0b724710e9ebf23a6 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:07.592823 31472 leader_election.cc:302] T 78f06bc68b954dc4b55b3c8c3c612c2f P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, a44f41b0b6644fb6aa29476aed8c964f; no voters: 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:07.593204 6024 raft_consensus.cc:2781] T 78f06bc68b954dc4b55b3c8c3c612c2f P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:27:07.593268 6024 raft_consensus.cc:482] T 78f06bc68b954dc4b55b3c8c3c612c2f P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.593320 6024 raft_consensus.cc:3037] T 78f06bc68b954dc4b55b3c8c3c612c2f P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Advancing to term 2
W20240502 21:27:07.595427 2161 leader_election.cc:334] T a8684b1ca3a642688a232afaf1b28f96 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
W20240502 21:27:07.595716 2161 leader_election.cc:334] T a8684b1ca3a642688a232afaf1b28f96 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.596118 6024 raft_consensus.cc:504] T 78f06bc68b954dc4b55b3c8c3c612c2f P a44f41b0b6644fb6aa29476aed8c964f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:07.596621 1831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8684b1ca3a642688a232afaf1b28f96" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:27:07.596818 1831 raft_consensus.cc:3037] T a8684b1ca3a642688a232afaf1b28f96 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:07.598322 6070 ts_tablet_manager.cc:1875] T 988a0a9425434cd0b724710e9ebf23a6 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 21:27:07.598621 31469 leader_election.cc:334] T 78f06bc68b954dc4b55b3c8c3c612c2f P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.599288 1831 raft_consensus.cc:2454] T a8684b1ca3a642688a232afaf1b28f96 P 1e04157b98dd4fad98858c290b73c3db [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 2.
I20240502 21:27:07.599390 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f06bc68b954dc4b55b3c8c3c612c2f" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:27:07.599532 31666 raft_consensus.cc:3037] T 78f06bc68b954dc4b55b3c8c3c612c2f P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:07.599640 2163 leader_election.cc:302] T a8684b1ca3a642688a232afaf1b28f96 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, d1406e92aec84ac585304e57bd37a40d; no voters: 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:07.599993 5661 raft_consensus.cc:2781] T a8684b1ca3a642688a232afaf1b28f96 P d1406e92aec84ac585304e57bd37a40d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:07.600118 5661 raft_consensus.cc:686] T a8684b1ca3a642688a232afaf1b28f96 P d1406e92aec84ac585304e57bd37a40d [term 2 LEADER]: Becoming Leader. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Running, Role: LEADER
I20240502 21:27:07.600324 5661 consensus_queue.cc:237] T a8684b1ca3a642688a232afaf1b28f96 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } }
I20240502 21:27:07.602149 31666 raft_consensus.cc:2454] T 78f06bc68b954dc4b55b3c8c3c612c2f P 05f4fac41c4f4959b9b316cb112d0ada [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 2.
I20240502 21:27:07.603245 31472 leader_election.cc:302] T 78f06bc68b954dc4b55b3c8c3c612c2f P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, a44f41b0b6644fb6aa29476aed8c964f; no voters: 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:07.605460 6027 raft_consensus.cc:482] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.605590 6027 raft_consensus.cc:504] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:27:07.606242 6027 leader_election.cc:288] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:27:07.612445 5910 raft_consensus.cc:2781] T 78f06bc68b954dc4b55b3c8c3c612c2f P a44f41b0b6644fb6aa29476aed8c964f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:07.612561 5910 raft_consensus.cc:686] T 78f06bc68b954dc4b55b3c8c3c612c2f P a44f41b0b6644fb6aa29476aed8c964f [term 2 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:27:07.612764 5910 consensus_queue.cc:237] T 78f06bc68b954dc4b55b3c8c3c612c2f P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:07.624853 6070 ts_tablet_manager.cc:1888] T 988a0a9425434cd0b724710e9ebf23a6 P 08d7a0680d244b55b93d17f9ab57b84c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:07.626647 5215 raft_consensus.cc:482] T c977ca41b72d4d45ac7146fbca350824 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.626793 5215 raft_consensus.cc:504] T c977ca41b72d4d45ac7146fbca350824 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:07.627517 5215 leader_election.cc:288] T c977ca41b72d4d45ac7146fbca350824 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:27:07.627926 31924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c977ca41b72d4d45ac7146fbca350824" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:27:07.628146 31924 raft_consensus.cc:2454] T c977ca41b72d4d45ac7146fbca350824 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:27:07.628823 1770 leader_election.cc:302] T c977ca41b72d4d45ac7146fbca350824 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:27:07.629268 5215 raft_consensus.cc:2781] T c977ca41b72d4d45ac7146fbca350824 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:27:07.629384 5215 raft_consensus.cc:482] T c977ca41b72d4d45ac7146fbca350824 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.629503 5215 raft_consensus.cc:3037] T c977ca41b72d4d45ac7146fbca350824 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:07.630895 5954 raft_consensus.cc:504] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } }
I20240502 21:27:07.631904 5215 raft_consensus.cc:504] T c977ca41b72d4d45ac7146fbca350824 P 1e04157b98dd4fad98858c290b73c3db [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:07.632355 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02f83ec0a494da5b66241d9b379a03f" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:27:07.632572 31535 raft_consensus.cc:3037] T f02f83ec0a494da5b66241d9b379a03f P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:07.633292 31924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c977ca41b72d4d45ac7146fbca350824" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:27:07.633595 31924 raft_consensus.cc:3037] T c977ca41b72d4d45ac7146fbca350824 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:07.635003 6070 log.cc:1196] T 988a0a9425434cd0b724710e9ebf23a6 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/wals/988a0a9425434cd0b724710e9ebf23a6
I20240502 21:27:07.636656 31924 raft_consensus.cc:2454] T c977ca41b72d4d45ac7146fbca350824 P 2d50f33557944bb480ea32199d9ee4f1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 2.
I20240502 21:27:07.636760 31381 catalog_manager.cc:4830] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): tablet 988a0a9425434cd0b724710e9ebf23a6 (table rebalance_test_table_9 [id=5eee55eb24e94c988b4ddbfe2f8442d3]) successfully deleted
I20240502 21:27:07.637034 1770 leader_election.cc:302] T c977ca41b72d4d45ac7146fbca350824 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:27:07.637346 5854 raft_consensus.cc:2781] T c977ca41b72d4d45ac7146fbca350824 P 1e04157b98dd4fad98858c290b73c3db [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:07.637382 5910 raft_consensus.cc:482] T 4d5d4121e8284195b76bfd12f826f942 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.637497 5910 raft_consensus.cc:504] T 4d5d4121e8284195b76bfd12f826f942 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:07.638101 5910 leader_election.cc:288] T 4d5d4121e8284195b76bfd12f826f942 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:27:07.638072 6003 raft_consensus.cc:482] T 4d5d4121e8284195b76bfd12f826f942 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.638196 6003 raft_consensus.cc:504] T 4d5d4121e8284195b76bfd12f826f942 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:07.638773 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d5d4121e8284195b76bfd12f826f942" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:27:07.638962 31797 raft_consensus.cc:2454] T 4d5d4121e8284195b76bfd12f826f942 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:27:07.639207 6003 leader_election.cc:288] T 4d5d4121e8284195b76bfd12f826f942 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:27:07.639389 31471 leader_election.cc:302] T 4d5d4121e8284195b76bfd12f826f942 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:27:07.639884 6013 raft_consensus.cc:2781] T 4d5d4121e8284195b76bfd12f826f942 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:27:07.639945 6013 raft_consensus.cc:482] T 4d5d4121e8284195b76bfd12f826f942 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.640003 6013 raft_consensus.cc:3037] T 4d5d4121e8284195b76bfd12f826f942 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:07.640473 31536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d5d4121e8284195b76bfd12f826f942" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
W20240502 21:27:07.642055 31730 leader_election.cc:334] T 4d5d4121e8284195b76bfd12f826f942 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.644196 5954 leader_election.cc:288] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 election: Requested vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:27:07.646085 31394 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_extra_table_48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 21:27:07.646903 31469 leader_election.cc:334] T 4d5d4121e8284195b76bfd12f826f942 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
W20240502 21:27:07.648308 1767 leader_election.cc:334] T c977ca41b72d4d45ac7146fbca350824 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
W20240502 21:27:07.649091 1767 leader_election.cc:334] T c977ca41b72d4d45ac7146fbca350824 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.651172 5215 leader_election.cc:288] T c977ca41b72d4d45ac7146fbca350824 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 2 election: Requested vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:27:07.653136 31535 raft_consensus.cc:2454] T f02f83ec0a494da5b66241d9b379a03f P a44f41b0b6644fb6aa29476aed8c964f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 2.
I20240502 21:27:07.653569 2033 leader_election.cc:302] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:27:07.652302 6013 raft_consensus.cc:504] T 4d5d4121e8284195b76bfd12f826f942 P a44f41b0b6644fb6aa29476aed8c964f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:07.654285 6013 leader_election.cc:288] T 4d5d4121e8284195b76bfd12f826f942 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 2 election: Requested vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:27:07.654767 31536 raft_consensus.cc:2379] T 4d5d4121e8284195b76bfd12f826f942 P a44f41b0b6644fb6aa29476aed8c964f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 08d7a0680d244b55b93d17f9ab57b84c in current term 2: Already voted for candidate a44f41b0b6644fb6aa29476aed8c964f in this term.
I20240502 21:27:07.654821 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d5d4121e8284195b76bfd12f826f942" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:27:07.655040 31797 raft_consensus.cc:3037] T 4d5d4121e8284195b76bfd12f826f942 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:07.655560 31733 leader_election.cc:302] T 4d5d4121e8284195b76bfd12f826f942 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c; no voters: 9dec645b473f40578e726f3e50e219bf, a44f41b0b6644fb6aa29476aed8c964f
I20240502 21:27:07.657488 31797 raft_consensus.cc:2454] T 4d5d4121e8284195b76bfd12f826f942 P 08d7a0680d244b55b93d17f9ab57b84c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 2.
I20240502 21:27:07.659005 6003 raft_consensus.cc:2726] T 4d5d4121e8284195b76bfd12f826f942 P 08d7a0680d244b55b93d17f9ab57b84c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:27:07.659848 31471 leader_election.cc:302] T 4d5d4121e8284195b76bfd12f826f942 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:27:07.660089 31906 tablet_service.cc:1467] Processing CreateTablet for tablet 8702f1a38b274f08affe558f6ef62d74 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=4cb9323058244212a2712a39dfca172b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:07.660513 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8702f1a38b274f08affe558f6ef62d74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.660915 6013 raft_consensus.cc:2781] T 4d5d4121e8284195b76bfd12f826f942 P a44f41b0b6644fb6aa29476aed8c964f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:07.661049 6013 raft_consensus.cc:686] T 4d5d4121e8284195b76bfd12f826f942 P a44f41b0b6644fb6aa29476aed8c964f [term 2 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:27:07.661521 6013 consensus_queue.cc:237] T 4d5d4121e8284195b76bfd12f826f942 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:07.662823 31775 tablet_service.cc:1467] Processing CreateTablet for tablet 8702f1a38b274f08affe558f6ef62d74 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=4cb9323058244212a2712a39dfca172b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:07.663220 31775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8702f1a38b274f08affe558f6ef62d74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.663146 31905 tablet_service.cc:1467] Processing CreateTablet for tablet e2ed57eb07384dc8953715c2c71b78b8 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=4cb9323058244212a2712a39dfca172b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:07.663506 31905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2ed57eb07384dc8953715c2c71b78b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.663691 31514 tablet_service.cc:1467] Processing CreateTablet for tablet e2ed57eb07384dc8953715c2c71b78b8 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=4cb9323058244212a2712a39dfca172b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:07.664016 31514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2ed57eb07384dc8953715c2c71b78b8. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 21:27:07.664973 31469 leader_election.cc:334] T 4d5d4121e8284195b76bfd12f826f942 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.665079 31774 tablet_service.cc:1467] Processing CreateTablet for tablet 7b7b92c3d32844a99a9efe8a5bcc5a3d (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=4cb9323058244212a2712a39dfca172b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:07.665431 31774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b7b92c3d32844a99a9efe8a5bcc5a3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.668876 2208 tablet_service.cc:1467] Processing CreateTablet for tablet 8702f1a38b274f08affe558f6ef62d74 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=4cb9323058244212a2712a39dfca172b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:07.669274 2208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8702f1a38b274f08affe558f6ef62d74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.669724 5960 raft_consensus.cc:482] T 565f30d5539c4b7ea956dc49d0f9a55e P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.669906 5960 raft_consensus.cc:504] T 565f30d5539c4b7ea956dc49d0f9a55e P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:07.670706 5960 leader_election.cc:288] T 565f30d5539c4b7ea956dc49d0f9a55e P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:27:07.671155 5960 raft_consensus.cc:2781] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:07.671375 5960 raft_consensus.cc:686] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b [term 2 LEADER]: Becoming Leader. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Running, Role: LEADER
I20240502 21:27:07.671666 5960 consensus_queue.cc:237] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } }
I20240502 21:27:07.672755 2210 tablet_service.cc:1467] Processing CreateTablet for tablet e2ed57eb07384dc8953715c2c71b78b8 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=4cb9323058244212a2712a39dfca172b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:07.673063 2210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2ed57eb07384dc8953715c2c71b78b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.673368 5215 raft_consensus.cc:482] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.673528 5215 raft_consensus.cc:504] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } }
I20240502 21:27:07.674716 5215 leader_election.cc:288] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:27:07.676898 31645 tablet_service.cc:1467] Processing CreateTablet for tablet 7b7b92c3d32844a99a9efe8a5bcc5a3d (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=4cb9323058244212a2712a39dfca172b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:07.677258 31645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b7b92c3d32844a99a9efe8a5bcc5a3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:07.678086 6036 tablet_bootstrap.cc:492] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:27:07.679327 6036 tablet_bootstrap.cc:654] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
W20240502 21:27:07.684401 1767 leader_election.cc:334] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.685045 6083 tablet_bootstrap.cc:492] T 8702f1a38b274f08affe558f6ef62d74 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:27:07.686677 6024 leader_election.cc:288] T 78f06bc68b954dc4b55b3c8c3c612c2f P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 2 election: Requested vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:27:07.686839 6083 tablet_bootstrap.cc:654] T 8702f1a38b274f08affe558f6ef62d74 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:07.688133 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f06bc68b954dc4b55b3c8c3c612c2f" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:27:07.690486 31924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b407c2c9b3a14ce3ada52b7856c6ecee" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:27:07.690654 31924 raft_consensus.cc:2454] T b407c2c9b3a14ce3ada52b7856c6ecee P 2d50f33557944bb480ea32199d9ee4f1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 2.
W20240502 21:27:07.691213 1767 consensus_peers.cc:485] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:07.696378 6083 tablet_bootstrap.cc:492] T 8702f1a38b274f08affe558f6ef62d74 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:27:07.696588 6083 ts_tablet_manager.cc:1367] T 8702f1a38b274f08affe558f6ef62d74 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.005s
I20240502 21:27:07.696705 2096 raft_consensus.cc:1261] T 04c07a003bfe472e9142df6082759ae1 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:07.687834 1834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13c3b9c4b7c471fa34156b781473ec4" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:27:07.697459 1834 raft_consensus.cc:2454] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 2.
I20240502 21:27:07.697463 6083 raft_consensus.cc:348] T 8702f1a38b274f08affe558f6ef62d74 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:07.698391 2096 raft_consensus.cc:1261] T 64c31d44d0224d54a306a29f7a11f568 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:07.698429 6083 raft_consensus.cc:374] T 8702f1a38b274f08affe558f6ef62d74 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:07.698650 6083 raft_consensus.cc:729] T 8702f1a38b274f08affe558f6ef62d74 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:27:07.698978 6083 consensus_queue.cc:260] T 8702f1a38b274f08affe558f6ef62d74 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:07.699476 5215 consensus_queue.cc:1035] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:07.700104 6083 ts_tablet_manager.cc:1398] T 8702f1a38b274f08affe558f6ef62d74 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 21:27:07.700753 6083 tablet_bootstrap.cc:492] T e2ed57eb07384dc8953715c2c71b78b8 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:27:07.702152 6083 tablet_bootstrap.cc:654] T e2ed57eb07384dc8953715c2c71b78b8 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:07.706549 31388 catalog_manager.cc:5484] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: term changed from 1 to 2, leader changed from 9dec645b473f40578e726f3e50e219bf (127.24.226.134) to 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135). New cstate: current_term: 2 leader_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:07.710500 6027 raft_consensus.cc:482] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.710633 6027 raft_consensus.cc:504] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } }
I20240502 21:27:07.711323 6027 leader_election.cc:288] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:27:07.711323 31797 raft_consensus.cc:1261] T 64c31d44d0224d54a306a29f7a11f568 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:07.711910 31603 leader_election.cc:302] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:27:07.712134 6027 raft_consensus.cc:2781] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 21:27:07.712455 6027 raft_consensus.cc:482] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.712546 6027 raft_consensus.cc:3037] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:07.714641 6085 tablet_bootstrap.cc:492] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:27:07.716553 5215 consensus_queue.cc:1035] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:07.714259 6088 tablet_bootstrap.cc:492] T e2ed57eb07384dc8953715c2c71b78b8 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:27:07.719856 6088 tablet_bootstrap.cc:654] T e2ed57eb07384dc8953715c2c71b78b8 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:07.721498 6085 tablet_bootstrap.cc:654] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:07.716257 6027 raft_consensus.cc:504] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada [term 3 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:27:07.727272 31471 leader_election.cc:302] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, a44f41b0b6644fb6aa29476aed8c964f; no voters: 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:07.728928 6024 raft_consensus.cc:2781] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 21:27:07.729008 6024 raft_consensus.cc:482] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.729081 6024 raft_consensus.cc:3037] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:07.732858 6024 raft_consensus.cc:504] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } }
I20240502 21:27:07.733826 31386 catalog_manager.cc:5484] T 78f06bc68b954dc4b55b3c8c3c612c2f P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 1 to 2, leader changed from 9dec645b473f40578e726f3e50e219bf (127.24.226.134) to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 2 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } }
W20240502 21:27:07.734284 31379 catalog_manager.cc:4573] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): CreateTablet RPC failed for tablet 7b7b92c3d32844a99a9efe8a5bcc5a3d: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.744172 6085 tablet_bootstrap.cc:492] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:27:07.744474 6021 consensus_queue.cc:1035] T eed5a5c41d9a46ce8a712b5b399e0472 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:07.746663 6085 ts_tablet_manager.cc:1367] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.000s
I20240502 21:27:07.744704 31603 leader_election.cc:302] T 78f06bc68b954dc4b55b3c8c3c612c2f P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada; no voters: 9dec645b473f40578e726f3e50e219bf, a44f41b0b6644fb6aa29476aed8c964f
I20240502 21:27:07.748152 6021 raft_consensus.cc:2726] T 78f06bc68b954dc4b55b3c8c3c612c2f P 05f4fac41c4f4959b9b316cb112d0ada [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:27:07.744990 31600 leader_election.cc:334] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.746119 6027 leader_election.cc:288] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 3 election: Requested vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:27:07.749691 6085 raft_consensus.cc:348] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:07.751535 6085 raft_consensus.cc:374] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:07.751835 6085 raft_consensus.cc:729] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:27:07.752254 6085 consensus_queue.cc:260] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:07.752648 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b407c2c9b3a14ce3ada52b7856c6ecee" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:27:07.752813 31925 raft_consensus.cc:3037] T b407c2c9b3a14ce3ada52b7856c6ecee P 2d50f33557944bb480ea32199d9ee4f1 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:07.753634 2224 raft_consensus.cc:1261] T 51551a1967b74094b373cc030e2bf850 P d1406e92aec84ac585304e57bd37a40d [term 2 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 21:27:07.755867 31925 raft_consensus.cc:2454] T b407c2c9b3a14ce3ada52b7856c6ecee P 2d50f33557944bb480ea32199d9ee4f1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 3.
I20240502 21:27:07.758682 31386 catalog_manager.cc:5484] T 69432f98b88740adb4fcbc112298cf20 P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: term changed from 1 to 2, leader changed from 9dec645b473f40578e726f3e50e219bf (127.24.226.134) to 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130). New cstate: current_term: 2 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } }
W20240502 21:27:07.762852 31600 leader_election.cc:334] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.767346 6085 ts_tablet_manager.cc:1398] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.020s user 0.000s sys 0.002s
I20240502 21:27:07.768471 31388 catalog_manager.cc:5484] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: term changed from 2 to 3, leader changed from 9dec645b473f40578e726f3e50e219bf (127.24.226.134) to 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132). New cstate: current_term: 3 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:07.781021 5854 raft_consensus.cc:686] T c977ca41b72d4d45ac7146fbca350824 P 1e04157b98dd4fad98858c290b73c3db [term 2 LEADER]: Becoming Leader. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Running, Role: LEADER
I20240502 21:27:07.787006 31388 catalog_manager.cc:5484] T 4d5d4121e8284195b76bfd12f826f942 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 1 to 2, leader changed from 9dec645b473f40578e726f3e50e219bf (127.24.226.134) to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 2 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } }
W20240502 21:27:07.791498 31600 leader_election.cc:334] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.697348 5643 consensus_queue.cc:1035] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:07.801311 5854 consensus_queue.cc:237] T c977ca41b72d4d45ac7146fbca350824 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:07.839666 31603 leader_election.cc:302] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 2d50f33557944bb480ea32199d9ee4f1; no voters: 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:07.844642 6027 raft_consensus.cc:2781] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:07.845144 6027 raft_consensus.cc:686] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada [term 3 LEADER]: Becoming Leader. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Running, Role: LEADER
I20240502 21:27:07.845568 6027 consensus_queue.cc:237] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:27:07.750183 6088 tablet_bootstrap.cc:492] T e2ed57eb07384dc8953715c2c71b78b8 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:27:07.752555 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b20c2bb60f6149458a62ee443b8cf3ab" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:27:07.848325 31535 raft_consensus.cc:2454] T b20c2bb60f6149458a62ee443b8cf3ab P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:27:07.848357 6088 ts_tablet_manager.cc:1367] T e2ed57eb07384dc8953715c2c71b78b8 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.134s user 0.007s sys 0.000s
I20240502 21:27:07.754132 5705 consensus_queue.cc:1035] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:07.781858 31533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3bdc518b84b5abc362cc4e12027c3" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:27:07.848699 31533 raft_consensus.cc:2454] T acb3bdc518b84b5abc362cc4e12027c3 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 1.
I20240502 21:27:07.849232 6088 raft_consensus.cc:348] T e2ed57eb07384dc8953715c2c71b78b8 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:07.849440 6088 raft_consensus.cc:374] T e2ed57eb07384dc8953715c2c71b78b8 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:07.849494 6088 raft_consensus.cc:729] T e2ed57eb07384dc8953715c2c71b78b8 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:27:07.849684 6088 consensus_queue.cc:260] T e2ed57eb07384dc8953715c2c71b78b8 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:07.850179 6088 ts_tablet_manager.cc:1398] T e2ed57eb07384dc8953715c2c71b78b8 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20240502 21:27:07.859561 31860 consensus_peers.cc:485] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:07.860688 6024 leader_election.cc:288] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 3 election: Requested vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
W20240502 21:27:07.862279 31469 leader_election.cc:334] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:07.862931 1834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13c3b9c4b7c471fa34156b781473ec4" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:27:07.863184 1834 raft_consensus.cc:3037] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:07.866214 1834 raft_consensus.cc:2454] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 3.
I20240502 21:27:07.869376 31471 leader_election.cc:302] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, a44f41b0b6644fb6aa29476aed8c964f; no voters: 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:07.881071 5896 raft_consensus.cc:482] T 8702f1a38b274f08affe558f6ef62d74 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:07.881220 5896 raft_consensus.cc:504] T 8702f1a38b274f08affe558f6ef62d74 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:07.881829 5896 leader_election.cc:288] T 8702f1a38b274f08affe558f6ef62d74 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:27:07.882581 6013 raft_consensus.cc:2781] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:07.882802 6013 raft_consensus.cc:686] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f [term 3 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:27:07.883224 6013 consensus_queue.cc:237] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } }
I20240502 21:27:07.889304 31793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8702f1a38b274f08affe558f6ef62d74" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
W20240502 21:27:07.889997 31862 leader_election.cc:341] T 8702f1a38b274f08affe558f6ef62d74 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:07.905061 2224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8702f1a38b274f08affe558f6ef62d74" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
W20240502 21:27:07.907615 31862 leader_election.cc:341] T 8702f1a38b274f08affe558f6ef62d74 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:07.907744 31862 leader_election.cc:302] T 8702f1a38b274f08affe558f6ef62d74 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1; no voters: 08d7a0680d244b55b93d17f9ab57b84c, d1406e92aec84ac585304e57bd37a40d
I20240502 21:27:07.908206 5896 raft_consensus.cc:2726] T 8702f1a38b274f08affe558f6ef62d74 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 21:27:07.911201 31469 consensus_peers.cc:485] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:07.922561 31860 consensus_peers.cc:485] T 41029f1332c64ba58ee0ee45714ba2a8 P 2d50f33557944bb480ea32199d9ee4f1 -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:07.840021 31603 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.226.132:35933, user_credentials={real_user=slave}} blocked reactor thread for 61608.8us
I20240502 21:27:07.933326 31603 leader_election.cc:302] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, a44f41b0b6644fb6aa29476aed8c964f; no voters: 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:07.933650 5714 raft_consensus.cc:2781] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:27:07.933728 5714 raft_consensus.cc:482] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:07.933782 5714 raft_consensus.cc:3037] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:07.937170 5714 raft_consensus.cc:504] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } }
I20240502 21:27:07.938066 5714 leader_election.cc:288] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 2 election: Requested vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:27:07.940774 5714 raft_consensus.cc:482] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:07.940941 5714 raft_consensus.cc:504] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:07.941918 5714 leader_election.cc:288] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:27:07.942028 31793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b7b92c3d32844a99a9efe8a5bcc5a3d" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:27:07.942174 5705 raft_consensus.cc:482] T e2ed57eb07384dc8953715c2c71b78b8 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 21:27:07.942524 31602 leader_election.cc:341] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:07.942744 5705 raft_consensus.cc:504] T e2ed57eb07384dc8953715c2c71b78b8 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:07.943918 5705 leader_election.cc:288] T e2ed57eb07384dc8953715c2c71b78b8 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:27:07.944244 2225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ed57eb07384dc8953715c2c71b78b8" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
W20240502 21:27:07.944846 31471 leader_election.cc:341] T e2ed57eb07384dc8953715c2c71b78b8 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:07.945487 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ed57eb07384dc8953715c2c71b78b8" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
W20240502 21:27:07.946063 31472 leader_election.cc:341] T e2ed57eb07384dc8953715c2c71b78b8 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:07.946225 31472 leader_election.cc:302] T e2ed57eb07384dc8953715c2c71b78b8 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a44f41b0b6644fb6aa29476aed8c964f; no voters: 2d50f33557944bb480ea32199d9ee4f1, d1406e92aec84ac585304e57bd37a40d
I20240502 21:27:07.946763 5705 raft_consensus.cc:2726] T e2ed57eb07384dc8953715c2c71b78b8 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:27:07.955799 6086 tablet_bootstrap.cc:492] T e2ed57eb07384dc8953715c2c71b78b8 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:27:07.964510 31388 catalog_manager.cc:5484] T a8684b1ca3a642688a232afaf1b28f96 P d1406e92aec84ac585304e57bd37a40d reported cstate change: term changed from 1 to 2, leader changed from 9dec645b473f40578e726f3e50e219bf (127.24.226.134) to d1406e92aec84ac585304e57bd37a40d (127.24.226.136). New cstate: current_term: 2 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:07.964833 31925 raft_consensus.cc:1261] T 5a54d2aad63c4f3182992f00b8564acf P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:07.965118 1834 raft_consensus.cc:1261] T 5a54d2aad63c4f3182992f00b8564acf P 1e04157b98dd4fad98858c290b73c3db [term 1 LEARNER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:07.965523 6003 consensus_queue.cc:1035] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:07.965821 6100 consensus_queue.cc:1035] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:07.968235 2092 raft_consensus.cc:1261] T 41029f1332c64ba58ee0ee45714ba2a8 P 1ade0354c44c4c37ae480c7a94b11e1b [term 2 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:27:07.972983 2225 raft_consensus.cc:1261] T 5a54d2aad63c4f3182992f00b8564acf P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Refusing update from remote peer 08d7a0680d244b55b93d17f9ab57b84c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:07.973657 6003 consensus_queue.cc:1035] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:07.977985 31386 catalog_manager.cc:5576] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807) does not have the latest schema for tablet 64c31d44d0224d54a306a29f7a11f568 (table rebalancer_dynamic_table [id=0ae9080ead2341c3bac35ab05c4395f1]). Expected version 1 got 0
I20240502 21:27:07.982995 6036 tablet_bootstrap.cc:492] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
W20240502 21:27:07.979507 2032 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.226.133:39863, user_credentials={real_user=slave}} blocked reactor thread for 375394us
I20240502 21:27:07.984054 2224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "565f30d5539c4b7ea956dc49d0f9a55e" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:27:07.984395 2224 raft_consensus.cc:2454] T 565f30d5539c4b7ea956dc49d0f9a55e P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 1.
I20240502 21:27:07.986099 31665 raft_consensus.cc:1261] T f9bbc56ba2f841b9bb36a864e1a1f808 P 05f4fac41c4f4959b9b316cb112d0ada [term 3 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
W20240502 21:27:07.988531 31603 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.226.129:41039, user_credentials={real_user=slave}} blocked reactor thread for 55228.8us
I20240502 21:27:07.990410 31386 catalog_manager.cc:5576] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807) does not have the latest schema for tablet 64c31d44d0224d54a306a29f7a11f568 (table rebalancer_dynamic_table [id=0ae9080ead2341c3bac35ab05c4395f1]). Expected version 1 got 0
W20240502 21:27:07.994268 1770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.226.132:35933, user_credentials={real_user=slave}} blocked reactor thread for 203527us
I20240502 21:27:07.996285 6036 ts_tablet_manager.cc:1367] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.318s user 0.005s sys 0.007s
I20240502 21:27:07.997130 6036 raft_consensus.cc:348] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:07.997480 31386 catalog_manager.cc:5484] T c977ca41b72d4d45ac7146fbca350824 P 1e04157b98dd4fad98858c290b73c3db reported cstate change: term changed from 1 to 2, leader changed from 9dec645b473f40578e726f3e50e219bf (127.24.226.134) to 1e04157b98dd4fad98858c290b73c3db (127.24.226.133). New cstate: current_term: 2 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:08.002063 1770 leader_election.cc:302] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, a44f41b0b6644fb6aa29476aed8c964f; no voters: 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:08.004812 1833 raft_consensus.cc:2932] T 5a54d2aad63c4f3182992f00b8564acf P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:27:08.011574 5215 raft_consensus.cc:2781] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:27:08.011668 5215 raft_consensus.cc:482] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:08.011755 5215 raft_consensus.cc:3037] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:08.014278 5215 raft_consensus.cc:504] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } }
I20240502 21:27:08.014830 5215 leader_election.cc:288] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 2 election: Requested vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:27:08.015288 6111 tablet.cc:1722] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:08.015872 6111 tablet.cc:1620] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db: MemRowSet was empty: no flush needed.
I20240502 21:27:08.017238 2224 raft_consensus.cc:2932] T 5a54d2aad63c4f3182992f00b8564acf P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:27:08.019862 31665 raft_consensus.cc:1261] T 04c07a003bfe472e9142df6082759ae1 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEARNER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:08.021603 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b20c2bb60f6149458a62ee443b8cf3ab" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:27:08.021992 31535 raft_consensus.cc:3037] T b20c2bb60f6149458a62ee443b8cf3ab P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:08.022715 31533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3bdc518b84b5abc362cc4e12027c3" candidate_uuid: "1e04157b98dd4fad98858c290b73c3db" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:27:08.023088 31533 raft_consensus.cc:3037] T acb3bdc518b84b5abc362cc4e12027c3 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:08.034328 31533 raft_consensus.cc:2454] T acb3bdc518b84b5abc362cc4e12027c3 P a44f41b0b6644fb6aa29476aed8c964f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04157b98dd4fad98858c290b73c3db in term 2.
I20240502 21:27:08.034804 1770 leader_election.cc:302] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:27:08.035899 5643 raft_consensus.cc:2781] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:08.036055 5643 raft_consensus.cc:686] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db [term 2 LEADER]: Becoming Leader. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Running, Role: LEADER
I20240502 21:27:08.036211 31535 raft_consensus.cc:2454] T b20c2bb60f6149458a62ee443b8cf3ab P a44f41b0b6644fb6aa29476aed8c964f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 2.
I20240502 21:27:08.036352 5643 consensus_queue.cc:237] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } }
W20240502 21:27:08.043030 1767 leader_election.cc:334] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:08.046353 31386 catalog_manager.cc:5484] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: term changed from 1 to 2, leader changed from 9dec645b473f40578e726f3e50e219bf (127.24.226.134) to 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135). New cstate: current_term: 2 leader_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:08.047013 31386 catalog_manager.cc:5576] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569) does not have the latest schema for tablet 64c31d44d0224d54a306a29f7a11f568 (table rebalancer_dynamic_table [id=0ae9080ead2341c3bac35ab05c4395f1]). Expected version 1 got 0
I20240502 21:27:08.047286 5896 consensus_queue.cc:1035] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 21:27:08.049485 31600 leader_election.cc:334] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:08.049860 31394 catalog_manager.cc:5484] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 2 to 3, leader changed from 9dec645b473f40578e726f3e50e219bf (127.24.226.134) to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 3 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:08.071014 2032 leader_election.cc:302] T 565f30d5539c4b7ea956dc49d0f9a55e P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:27:08.071656 5945 raft_consensus.cc:2781] T 565f30d5539c4b7ea956dc49d0f9a55e P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:27:08.071789 5945 raft_consensus.cc:482] T 565f30d5539c4b7ea956dc49d0f9a55e P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9dec645b473f40578e726f3e50e219bf)
I20240502 21:27:08.071885 5945 raft_consensus.cc:3037] T 565f30d5539c4b7ea956dc49d0f9a55e P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:08.073196 31603 leader_election.cc:302] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, a44f41b0b6644fb6aa29476aed8c964f; no voters: 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:08.074208 6027 raft_consensus.cc:2781] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:08.074321 6027 raft_consensus.cc:686] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada [term 2 LEADER]: Becoming Leader. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Running, Role: LEADER
I20240502 21:27:08.074340 5945 raft_consensus.cc:504] T 565f30d5539c4b7ea956dc49d0f9a55e P 1ade0354c44c4c37ae480c7a94b11e1b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:08.074563 6027 consensus_queue.cc:237] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } }
I20240502 21:27:08.074898 5945 leader_election.cc:288] T 565f30d5539c4b7ea956dc49d0f9a55e P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 election: Requested vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:27:07.992372 6091 mvcc.cc:205] Tried to move back new op lower bound from 7023350692587913216 to 7023350690854150144. Current Snapshot: MvccSnapshot[applied={T|T < 7023350692587913216}]
I20240502 21:27:07.987332 6112 tablet.cc:1722] T 64c31d44d0224d54a306a29f7a11f568 P 08d7a0680d244b55b93d17f9ab57b84c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:08.077935 5816 consensus_queue.cc:1035] T 41029f1332c64ba58ee0ee45714ba2a8 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:08.078308 6036 raft_consensus.cc:374] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:08.078415 6112 tablet.cc:1620] T 64c31d44d0224d54a306a29f7a11f568 P 08d7a0680d244b55b93d17f9ab57b84c: MemRowSet was empty: no flush needed.
I20240502 21:27:08.003620 6003 raft_consensus.cc:2932] T 5a54d2aad63c4f3182992f00b8564acf P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:27:08.079942 6036 raft_consensus.cc:729] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:27:08.080188 6036 consensus_queue.cc:260] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:08.081820 6086 tablet_bootstrap.cc:654] T e2ed57eb07384dc8953715c2c71b78b8 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:08.083139 6036 ts_tablet_manager.cc:1398] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.087s user 0.002s sys 0.000s
I20240502 21:27:08.083410 6036 tablet_bootstrap.cc:492] T 8702f1a38b274f08affe558f6ef62d74 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:27:08.084584 6036 tablet_bootstrap.cc:654] T 8702f1a38b274f08affe558f6ef62d74 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
W20240502 21:27:08.085991 31600 leader_election.cc:334] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
W20240502 21:27:08.086061 1770 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.226.129:41039, user_credentials={real_user=slave}} blocked reactor thread for 51246.9us
I20240502 21:27:08.086315 31600 leader_election.cc:302] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada; no voters: 08d7a0680d244b55b93d17f9ab57b84c, 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:08.087357 5714 raft_consensus.cc:2726] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:27:08.092146 31388 catalog_manager.cc:5484] T 5a54d2aad63c4f3182992f00b8564acf P 1e04157b98dd4fad98858c290b73c3db reported cstate change: config changed from index 7 to 8, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:27:08.097316 31925 raft_consensus.cc:2932] T 5a54d2aad63c4f3182992f00b8564acf P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1e04157b98dd4fad98858c290b73c3db (127.24.226.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { replace: true } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } } }
I20240502 21:27:08.097836 5215 consensus_queue.cc:1035] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:08.100024 6086 tablet_bootstrap.cc:492] T e2ed57eb07384dc8953715c2c71b78b8 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:27:08.100872 6083 tablet_bootstrap.cc:492] T e2ed57eb07384dc8953715c2c71b78b8 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:27:08.101007 6083 ts_tablet_manager.cc:1367] T e2ed57eb07384dc8953715c2c71b78b8 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.400s user 0.036s sys 0.113s
I20240502 21:27:08.101543 6083 raft_consensus.cc:348] T e2ed57eb07384dc8953715c2c71b78b8 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:08.101867 6086 ts_tablet_manager.cc:1367] T e2ed57eb07384dc8953715c2c71b78b8 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.146s user 0.004s sys 0.000s
I20240502 21:27:08.101974 6083 raft_consensus.cc:374] T e2ed57eb07384dc8953715c2c71b78b8 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:08.102048 6083 raft_consensus.cc:729] T e2ed57eb07384dc8953715c2c71b78b8 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:27:08.102491 6121 tablet.cc:1722] T 64c31d44d0224d54a306a29f7a11f568 P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:08.103094 6121 tablet.cc:1620] T 64c31d44d0224d54a306a29f7a11f568 P 1ade0354c44c4c37ae480c7a94b11e1b: MemRowSet was empty: no flush needed.
I20240502 21:27:08.103243 6083 consensus_queue.cc:260] T e2ed57eb07384dc8953715c2c71b78b8 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:08.109911 6086 raft_consensus.cc:348] T e2ed57eb07384dc8953715c2c71b78b8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:08.110532 6083 ts_tablet_manager.cc:1398] T e2ed57eb07384dc8953715c2c71b78b8 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240502 21:27:08.110960 6036 tablet_bootstrap.cc:492] T 8702f1a38b274f08affe558f6ef62d74 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:27:08.111235 6036 ts_tablet_manager.cc:1367] T 8702f1a38b274f08affe558f6ef62d74 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.002s
I20240502 21:27:08.111371 6086 raft_consensus.cc:374] T e2ed57eb07384dc8953715c2c71b78b8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:08.111619 31386 catalog_manager.cc:5484] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: term changed from 2 to 3, leader changed from 9dec645b473f40578e726f3e50e219bf (127.24.226.134) to 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130). New cstate: current_term: 3 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:08.112082 6086 raft_consensus.cc:729] T e2ed57eb07384dc8953715c2c71b78b8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:27:08.112360 6086 consensus_queue.cc:260] T e2ed57eb07384dc8953715c2c71b78b8 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
W20240502 21:27:08.113044 31600 consensus_peers.cc:485] T 69432f98b88740adb4fcbc112298cf20 P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:08.113541 6086 ts_tablet_manager.cc:1398] T e2ed57eb07384dc8953715c2c71b78b8 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
W20240502 21:27:08.113655 2030 leader_election.cc:334] T 565f30d5539c4b7ea956dc49d0f9a55e P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:08.112965 6036 raft_consensus.cc:348] T 8702f1a38b274f08affe558f6ef62d74 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:08.114007 6036 raft_consensus.cc:374] T 8702f1a38b274f08affe558f6ef62d74 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:08.114219 6036 raft_consensus.cc:729] T 8702f1a38b274f08affe558f6ef62d74 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:27:08.114607 6086 tablet_bootstrap.cc:492] T 8702f1a38b274f08affe558f6ef62d74 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:27:08.114930 6036 consensus_queue.cc:260] T 8702f1a38b274f08affe558f6ef62d74 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:08.117213 6036 ts_tablet_manager.cc:1398] T 8702f1a38b274f08affe558f6ef62d74 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
W20240502 21:27:08.120394 31469 consensus_peers.cc:485] T 78f06bc68b954dc4b55b3c8c3c612c2f P a44f41b0b6644fb6aa29476aed8c964f -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:08.120784 6086 tablet_bootstrap.cc:654] T 8702f1a38b274f08affe558f6ef62d74 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:08.123533 31393 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=0ae9080ead2341c3bac35ab05c4395f1] alter complete (version 1)
I20240502 21:27:08.126011 31393 catalog_manager.cc:5484] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db reported cstate change: term changed from 1 to 2, leader changed from 9dec645b473f40578e726f3e50e219bf (127.24.226.134) to 1e04157b98dd4fad98858c290b73c3db (127.24.226.133). New cstate: current_term: 2 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 21:27:08.127543 2030 leader_election.cc:334] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:08.129750 1833 raft_consensus.cc:1261] T a8684b1ca3a642688a232afaf1b28f96 P 1e04157b98dd4fad98858c290b73c3db [term 2 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:27:08.130481 5661 consensus_queue.cc:1035] T a8684b1ca3a642688a232afaf1b28f96 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:08.146394 31389 catalog_manager.cc:5484] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: term changed from 1 to 2, leader changed from 9dec645b473f40578e726f3e50e219bf (127.24.226.134) to 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130). New cstate: current_term: 2 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:08.147045 31666 raft_consensus.cc:1261] T 78f06bc68b954dc4b55b3c8c3c612c2f P 05f4fac41c4f4959b9b316cb112d0ada [term 2 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:27:08.148557 6013 consensus_queue.cc:1035] T 78f06bc68b954dc4b55b3c8c3c612c2f P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:08.149811 6086 tablet_bootstrap.cc:492] T 8702f1a38b274f08affe558f6ef62d74 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:27:08.150238 6086 ts_tablet_manager.cc:1367] T 8702f1a38b274f08affe558f6ef62d74 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.036s user 0.006s sys 0.000s
I20240502 21:27:08.152837 6086 raft_consensus.cc:348] T 8702f1a38b274f08affe558f6ef62d74 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:08.153801 6086 raft_consensus.cc:374] T 8702f1a38b274f08affe558f6ef62d74 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:08.154021 6086 raft_consensus.cc:729] T 8702f1a38b274f08affe558f6ef62d74 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:27:08.154381 6086 consensus_queue.cc:260] T 8702f1a38b274f08affe558f6ef62d74 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:08.155587 6086 ts_tablet_manager.cc:1398] T 8702f1a38b274f08affe558f6ef62d74 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:08.165705 1834 raft_consensus.cc:1261] T 69432f98b88740adb4fcbc112298cf20 P 1e04157b98dd4fad98858c290b73c3db [term 2 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:27:08.166318 6072 consensus_queue.cc:1035] T 69432f98b88740adb4fcbc112298cf20 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:27:08.180161 31469 consensus_peers.cc:485] T 4d5d4121e8284195b76bfd12f826f942 P a44f41b0b6644fb6aa29476aed8c964f -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:08.181533 2033 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.226.130:37645, user_credentials={real_user=slave}} blocked reactor thread for 107786us
W20240502 21:27:08.184805 2161 consensus_peers.cc:485] T a8684b1ca3a642688a232afaf1b28f96 P d1406e92aec84ac585304e57bd37a40d -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:08.184590 6095 raft_consensus.cc:2932] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
W20240502 21:27:08.185556 2030 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.226.134:43753, user_credentials={real_user=slave}} blocked reactor thread for 57378.7us
I20240502 21:27:08.188938 2096 raft_consensus.cc:2932] T 04c07a003bfe472e9142df6082759ae1 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:27:08.193313 31797 raft_consensus.cc:1261] T 4d5d4121e8284195b76bfd12f826f942 P 08d7a0680d244b55b93d17f9ab57b84c [term 2 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:27:08.193109 31664 raft_consensus.cc:2932] T 04c07a003bfe472e9142df6082759ae1 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
I20240502 21:27:08.193845 6016 consensus_queue.cc:1035] T 4d5d4121e8284195b76bfd12f826f942 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:08.211827 5417 raft_consensus.cc:482] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:08.212011 5417 raft_consensus.cc:504] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:08.212181 31393 catalog_manager.cc:5484] T 04c07a003bfe472e9142df6082759ae1 P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: config changed from index 2 to 3, 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e04157b98dd4fad98858c290b73c3db" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } attrs { replace: true } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } } }
W20240502 21:27:08.212564 2032 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.226.136:34907, user_credentials={real_user=slave}} blocked reactor thread for 141231us
I20240502 21:27:08.212718 5417 leader_election.cc:288] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:27:08.213810 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b7b92c3d32844a99a9efe8a5bcc5a3d" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:27:08.213989 31666 raft_consensus.cc:2454] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
W20240502 21:27:08.214635 31730 leader_election.cc:334] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
W20240502 21:27:08.219491 2030 leader_election.cc:334] T 565f30d5539c4b7ea956dc49d0f9a55e P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
W20240502 21:27:08.222760 2030 consensus_peers.cc:485] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:08.223814 2225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "565f30d5539c4b7ea956dc49d0f9a55e" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:27:08.224015 2225 raft_consensus.cc:3037] T 565f30d5539c4b7ea956dc49d0f9a55e P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:08.226835 2225 raft_consensus.cc:2454] T 565f30d5539c4b7ea956dc49d0f9a55e P d1406e92aec84ac585304e57bd37a40d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 2.
I20240502 21:27:08.227181 31733 leader_election.cc:302] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 08d7a0680d244b55b93d17f9ab57b84c; no voters: 9dec645b473f40578e726f3e50e219bf
W20240502 21:27:08.227670 31600 consensus_peers.cc:462] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:08.227701 6004 raft_consensus.cc:2781] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:08.227799 6004 raft_consensus.cc:482] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:08.227885 6004 raft_consensus.cc:3037] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:08.228562 2224 raft_consensus.cc:1261] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P d1406e92aec84ac585304e57bd37a40d [term 2 FOLLOWER]: Refusing update from remote peer 1ade0354c44c4c37ae480c7a94b11e1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:27:08.230464 6004 raft_consensus.cc:504] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:08.231060 6004 leader_election.cc:288] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Requested vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
W20240502 21:27:08.232993 31730 leader_election.cc:334] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:08.233394 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b7b92c3d32844a99a9efe8a5bcc5a3d" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:27:08.233554 31666 raft_consensus.cc:3037] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:27:08.235678 1767 consensus_peers.cc:485] T c977ca41b72d4d45ac7146fbca350824 P 1e04157b98dd4fad98858c290b73c3db -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:08.236326 31666 raft_consensus.cc:2454] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:27:08.237716 31733 leader_election.cc:302] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 08d7a0680d244b55b93d17f9ab57b84c; no voters: 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:08.238173 6004 raft_consensus.cc:2781] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:08.238296 6004 raft_consensus.cc:686] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Becoming Leader. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Running, Role: LEADER
I20240502 21:27:08.238514 6004 consensus_queue.cc:237] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } }
I20240502 21:27:08.247573 2032 leader_election.cc:302] T 565f30d5539c4b7ea956dc49d0f9a55e P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, d1406e92aec84ac585304e57bd37a40d; no voters: 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:08.247857 5945 raft_consensus.cc:2781] T 565f30d5539c4b7ea956dc49d0f9a55e P 1ade0354c44c4c37ae480c7a94b11e1b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:08.247989 5945 raft_consensus.cc:686] T 565f30d5539c4b7ea956dc49d0f9a55e P 1ade0354c44c4c37ae480c7a94b11e1b [term 2 LEADER]: Becoming Leader. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Running, Role: LEADER
I20240502 21:27:08.248256 5945 consensus_queue.cc:237] T 565f30d5539c4b7ea956dc49d0f9a55e P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:08.256776 31389 catalog_manager.cc:5484] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: term changed from 0 to 1, leader changed from <none> to 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131). New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:08.257246 6118 consensus_queue.cc:1035] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:08.269233 31394 catalog_manager.cc:5484] T 565f30d5539c4b7ea956dc49d0f9a55e P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: term changed from 1 to 2, leader changed from 9dec645b473f40578e726f3e50e219bf (127.24.226.134) to 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135). New cstate: current_term: 2 leader_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:08.272634 31389 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:08.279235 31925 raft_consensus.cc:1261] T c977ca41b72d4d45ac7146fbca350824 P 2d50f33557944bb480ea32199d9ee4f1 [term 2 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:27:08.279976 6095 consensus_queue.cc:1035] T c977ca41b72d4d45ac7146fbca350824 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:08.280874 6111 tablet.cc:1722] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:08.281299 6111 tablet.cc:1620] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db: MemRowSet was empty: no flush needed.
I20240502 21:27:08.283282 6112 tablet.cc:1722] T 64c31d44d0224d54a306a29f7a11f568 P 08d7a0680d244b55b93d17f9ab57b84c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:08.283799 6112 tablet.cc:1620] T 64c31d44d0224d54a306a29f7a11f568 P 08d7a0680d244b55b93d17f9ab57b84c: MemRowSet was empty: no flush needed.
I20240502 21:27:08.287174 6121 tablet.cc:1722] T 64c31d44d0224d54a306a29f7a11f568 P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:08.287607 6121 tablet.cc:1620] T 64c31d44d0224d54a306a29f7a11f568 P 1ade0354c44c4c37ae480c7a94b11e1b: MemRowSet was empty: no flush needed.
I20240502 21:27:08.305915 31923 raft_consensus.cc:1261] T b407c2c9b3a14ce3ada52b7856c6ecee P 2d50f33557944bb480ea32199d9ee4f1 [term 3 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240502 21:27:08.306484 6072 consensus_queue.cc:1035] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:08.312685 31389 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=0ae9080ead2341c3bac35ab05c4395f1] alter complete (version 2)
W20240502 21:27:08.334259 31600 consensus_peers.cc:485] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:08.334259 31469 consensus_peers.cc:485] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:08.366940 2030 consensus_peers.cc:485] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:08.369429 31535 raft_consensus.cc:1261] T f02f83ec0a494da5b66241d9b379a03f P a44f41b0b6644fb6aa29476aed8c964f [term 2 FOLLOWER]: Refusing update from remote peer 1ade0354c44c4c37ae480c7a94b11e1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 21:27:08.370213 5954 consensus_queue.cc:1035] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:08.375723 1834 raft_consensus.cc:1261] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db [term 3 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240502 21:27:08.376258 6024 consensus_queue.cc:1035] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:08.391330 6120 raft_consensus.cc:482] T e2ed57eb07384dc8953715c2c71b78b8 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:08.391446 6120 raft_consensus.cc:504] T e2ed57eb07384dc8953715c2c71b78b8 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:08.391966 6120 leader_election.cc:288] T e2ed57eb07384dc8953715c2c71b78b8 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:27:08.392742 31533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ed57eb07384dc8953715c2c71b78b8" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:27:08.392905 31533 raft_consensus.cc:2454] T e2ed57eb07384dc8953715c2c71b78b8 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:27:08.393520 31863 leader_election.cc:302] T e2ed57eb07384dc8953715c2c71b78b8 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:27:08.393776 6120 raft_consensus.cc:2781] T e2ed57eb07384dc8953715c2c71b78b8 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:08.393836 6120 raft_consensus.cc:482] T e2ed57eb07384dc8953715c2c71b78b8 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:08.393879 6120 raft_consensus.cc:3037] T e2ed57eb07384dc8953715c2c71b78b8 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:08.404501 2225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ed57eb07384dc8953715c2c71b78b8" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:27:08.405037 2225 raft_consensus.cc:2454] T e2ed57eb07384dc8953715c2c71b78b8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:27:08.407054 6120 raft_consensus.cc:504] T e2ed57eb07384dc8953715c2c71b78b8 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:08.407816 2225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ed57eb07384dc8953715c2c71b78b8" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:27:08.407984 2225 raft_consensus.cc:3037] T e2ed57eb07384dc8953715c2c71b78b8 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:08.410831 2225 raft_consensus.cc:2454] T e2ed57eb07384dc8953715c2c71b78b8 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:27:08.411774 31862 leader_election.cc:302] T e2ed57eb07384dc8953715c2c71b78b8 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:27:08.412122 5512 raft_consensus.cc:2781] T e2ed57eb07384dc8953715c2c71b78b8 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:08.412266 5512 raft_consensus.cc:686] T e2ed57eb07384dc8953715c2c71b78b8 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Running, Role: LEADER
I20240502 21:27:08.412669 6120 leader_election.cc:288] T e2ed57eb07384dc8953715c2c71b78b8 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Requested vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:27:08.412535 5512 consensus_queue.cc:237] T e2ed57eb07384dc8953715c2c71b78b8 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:08.413472 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ed57eb07384dc8953715c2c71b78b8" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:27:08.413633 31535 raft_consensus.cc:3037] T e2ed57eb07384dc8953715c2c71b78b8 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:08.417207 31535 raft_consensus.cc:2454] T e2ed57eb07384dc8953715c2c71b78b8 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:27:08.418493 31389 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:08.421451 31394 catalog_manager.cc:5484] T e2ed57eb07384dc8953715c2c71b78b8 P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132). New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:08.431934 6111 tablet.cc:1722] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:08.432082 6112 tablet.cc:1722] T 64c31d44d0224d54a306a29f7a11f568 P 08d7a0680d244b55b93d17f9ab57b84c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:08.438925 6121 tablet.cc:1722] T 64c31d44d0224d54a306a29f7a11f568 P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:08.440833 31393 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table_new_name [id=0ae9080ead2341c3bac35ab05c4395f1] alter complete (version 3)
I20240502 21:27:08.474978 31535 raft_consensus.cc:1261] T acb3bdc518b84b5abc362cc4e12027c3 P a44f41b0b6644fb6aa29476aed8c964f [term 2 FOLLOWER]: Refusing update from remote peer 1e04157b98dd4fad98858c290b73c3db: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 21:27:08.476552 5215 consensus_queue.cc:1035] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 21:27:08.478420 1767 consensus_peers.cc:485] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:08.509657 31533 raft_consensus.cc:1261] T b20c2bb60f6149458a62ee443b8cf3ab P a44f41b0b6644fb6aa29476aed8c964f [term 2 FOLLOWER]: Refusing update from remote peer 05f4fac41c4f4959b9b316cb112d0ada: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 21:27:08.510133 6021 consensus_queue.cc:1035] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 21:27:08.533742 31600 consensus_peers.cc:485] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:08.554922 31388 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:08.555153 31388 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:08.558605 31645 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:08.558933 31388 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 64c31d44d0224d54a306a29f7a11f568
I20240502 21:27:08.559823 31775 tablet_service.cc:1514] Processing DeleteTablet for tablet 64c31d44d0224d54a306a29f7a11f568 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:08 UTC) from {username='slave'} at 127.0.0.1:51596
I20240502 21:27:08.560139 1814 tablet_service.cc:1514] Processing DeleteTablet for tablet 64c31d44d0224d54a306a29f7a11f568 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:08 UTC) from {username='slave'} at 127.0.0.1:47510
I20240502 21:27:08.560485 2077 tablet_service.cc:1514] Processing DeleteTablet for tablet 64c31d44d0224d54a306a29f7a11f568 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:08 UTC) from {username='slave'} at 127.0.0.1:36740
I20240502 21:27:08.564016 6004 raft_consensus.cc:482] T 8702f1a38b274f08affe558f6ef62d74 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:08.564193 6004 raft_consensus.cc:504] T 8702f1a38b274f08affe558f6ef62d74 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:08.564890 6004 leader_election.cc:288] T 8702f1a38b274f08affe558f6ef62d74 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:27:08.565584 2224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8702f1a38b274f08affe558f6ef62d74" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:27:08.565804 2224 raft_consensus.cc:2454] T 8702f1a38b274f08affe558f6ef62d74 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:27:08.565783 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8702f1a38b274f08affe558f6ef62d74" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:27:08.565965 31925 raft_consensus.cc:2454] T 8702f1a38b274f08affe558f6ef62d74 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:27:08.566314 31732 leader_election.cc:302] T 8702f1a38b274f08affe558f6ef62d74 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:27:08.566754 5417 raft_consensus.cc:2781] T 8702f1a38b274f08affe558f6ef62d74 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:08.566836 5417 raft_consensus.cc:482] T 8702f1a38b274f08affe558f6ef62d74 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:08.566936 5417 raft_consensus.cc:3037] T 8702f1a38b274f08affe558f6ef62d74 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:08.569139 5417 raft_consensus.cc:504] T 8702f1a38b274f08affe558f6ef62d74 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:08.571921 2225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8702f1a38b274f08affe558f6ef62d74" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:27:08.572063 2225 raft_consensus.cc:3037] T 8702f1a38b274f08affe558f6ef62d74 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:08.574579 2225 raft_consensus.cc:2454] T 8702f1a38b274f08affe558f6ef62d74 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:27:08.575543 31732 leader_election.cc:302] T 8702f1a38b274f08affe558f6ef62d74 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:27:08.575877 6004 raft_consensus.cc:2781] T 8702f1a38b274f08affe558f6ef62d74 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:08.576031 6004 raft_consensus.cc:686] T 8702f1a38b274f08affe558f6ef62d74 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Becoming Leader. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Running, Role: LEADER
I20240502 21:27:08.576360 6004 consensus_queue.cc:237] T 8702f1a38b274f08affe558f6ef62d74 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:08.586439 6165 tablet_replica.cc:330] T 64c31d44d0224d54a306a29f7a11f568 P 1ade0354c44c4c37ae480c7a94b11e1b: stopping tablet replica
I20240502 21:27:08.586829 5417 leader_election.cc:288] T 8702f1a38b274f08affe558f6ef62d74 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Requested vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:27:08.587146 6165 raft_consensus.cc:2229] T 64c31d44d0224d54a306a29f7a11f568 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:08.587548 6165 raft_consensus.cc:2258] T 64c31d44d0224d54a306a29f7a11f568 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:08.588506 31388 catalog_manager.cc:5484] T 8702f1a38b274f08affe558f6ef62d74 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: term changed from 0 to 1, leader changed from <none> to 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131). New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:08.589335 6165 ts_tablet_manager.cc:1875] T 64c31d44d0224d54a306a29f7a11f568 P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:08.591056 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8702f1a38b274f08affe558f6ef62d74" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:27:08.591244 31925 raft_consensus.cc:3037] T 8702f1a38b274f08affe558f6ef62d74 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:08.593875 31925 raft_consensus.cc:2454] T 8702f1a38b274f08affe558f6ef62d74 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:27:08.595986 6165 ts_tablet_manager.cc:1888] T 64c31d44d0224d54a306a29f7a11f568 P 1ade0354c44c4c37ae480c7a94b11e1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:08.596784 6165 log.cc:1196] T 64c31d44d0224d54a306a29f7a11f568 P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal/wals/64c31d44d0224d54a306a29f7a11f568
I20240502 21:27:08.597447 6165 ts_tablet_manager.cc:1909] T 64c31d44d0224d54a306a29f7a11f568 P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting consensus metadata
I20240502 21:27:08.599617 6164 tablet_replica.cc:330] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db: stopping tablet replica
I20240502 21:27:08.604830 31381 catalog_manager.cc:4830] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): tablet 64c31d44d0224d54a306a29f7a11f568 (table rebalancer_dynamic_table_new_name [id=0ae9080ead2341c3bac35ab05c4395f1]) successfully deleted
I20240502 21:27:08.605659 31774 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 21:27:08.611076 6164 raft_consensus.cc:2229] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:08.611717 6164 raft_consensus.cc:2258] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:08.621241 6164 ts_tablet_manager.cc:1875] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:08.622116 6163 tablet_replica.cc:330] T 64c31d44d0224d54a306a29f7a11f568 P 08d7a0680d244b55b93d17f9ab57b84c: stopping tablet replica
I20240502 21:27:08.623020 6163 raft_consensus.cc:2229] T 64c31d44d0224d54a306a29f7a11f568 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:08.626157 6163 raft_consensus.cc:2258] T 64c31d44d0224d54a306a29f7a11f568 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:08.627573 6163 ts_tablet_manager.cc:1875] T 64c31d44d0224d54a306a29f7a11f568 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 21:27:08.631572 31600 consensus_peers.cc:462] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:08.634222 6163 ts_tablet_manager.cc:1888] T 64c31d44d0224d54a306a29f7a11f568 P 08d7a0680d244b55b93d17f9ab57b84c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:08.634449 6163 log.cc:1196] T 64c31d44d0224d54a306a29f7a11f568 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/wals/64c31d44d0224d54a306a29f7a11f568
I20240502 21:27:08.635246 6163 ts_tablet_manager.cc:1909] T 64c31d44d0224d54a306a29f7a11f568 P 08d7a0680d244b55b93d17f9ab57b84c: Deleting consensus metadata
I20240502 21:27:08.636646 31381 catalog_manager.cc:4830] TS 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): tablet 64c31d44d0224d54a306a29f7a11f568 (table rebalancer_dynamic_table_new_name [id=0ae9080ead2341c3bac35ab05c4395f1]) successfully deleted
W20240502 21:27:08.640547 31860 consensus_peers.cc:485] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:27:08.640739 31730 consensus_peers.cc:485] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 08d7a0680d244b55b93d17f9ab57b84c -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:08.643299 1814 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240502 21:27:08.643407 2161 consensus_peers.cc:485] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:27:08.648407 2077 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 21:27:08.665205 31388 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:08.667752 6164 ts_tablet_manager.cc:1888] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:08.667944 6164 log.cc:1196] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal/wals/64c31d44d0224d54a306a29f7a11f568
I20240502 21:27:08.668161 31906 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 21:27:08.668478 6164 ts_tablet_manager.cc:1909] T 64c31d44d0224d54a306a29f7a11f568 P 1e04157b98dd4fad98858c290b73c3db: Deleting consensus metadata
I20240502 21:27:08.671116 31381 catalog_manager.cc:4830] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): tablet 64c31d44d0224d54a306a29f7a11f568 (table rebalancer_dynamic_table_new_name [id=0ae9080ead2341c3bac35ab05c4395f1]) successfully deleted
I20240502 21:27:08.675201 1814 tablet_service.cc:1467] Processing CreateTablet for tablet 97a450582be947a48b4aa789389e9e5a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2dcceb9804094dee8643679132a7ba30]), partition=
I20240502 21:27:08.675542 1814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97a450582be947a48b4aa789389e9e5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:08.675544 2077 tablet_service.cc:1467] Processing CreateTablet for tablet 97a450582be947a48b4aa789389e9e5a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2dcceb9804094dee8643679132a7ba30]), partition=
I20240502 21:27:08.675840 2077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97a450582be947a48b4aa789389e9e5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:08.686908 6173 tablet_bootstrap.cc:492] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:27:08.687907 2210 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 21:27:08.688855 6173 tablet_bootstrap.cc:654] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
W20240502 21:27:08.702119 1767 consensus_peers.cc:485] T b67096e38f2e4c42a33e74f0145a04cc P 1e04157b98dd4fad98858c290b73c3db -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:27:08.708786 2161 consensus_peers.cc:485] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:27:08.717573 31514 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240502 21:27:08.730830 2161 consensus_peers.cc:485] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:27:08.733736 31469 consensus_peers.cc:485] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:27:08.734269 6172 tablet_bootstrap.cc:492] T 97a450582be947a48b4aa789389e9e5a P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
W20240502 21:27:08.734639 31379 catalog_manager.cc:4573] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): CreateTablet RPC failed for tablet 97a450582be947a48b4aa789389e9e5a: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:08.736586 6172 tablet_bootstrap.cc:654] T 97a450582be947a48b4aa789389e9e5a P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:08.753978 6004 consensus_queue.cc:1035] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:08.756059 6172 tablet_bootstrap.cc:492] T 97a450582be947a48b4aa789389e9e5a P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:27:08.756187 6172 ts_tablet_manager.cc:1367] T 97a450582be947a48b4aa789389e9e5a P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.003s
I20240502 21:27:08.756989 6172 raft_consensus.cc:348] T 97a450582be947a48b4aa789389e9e5a P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:08.757182 6172 raft_consensus.cc:374] T 97a450582be947a48b4aa789389e9e5a P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:08.757261 6172 raft_consensus.cc:729] T 97a450582be947a48b4aa789389e9e5a P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:27:08.757450 6172 consensus_queue.cc:260] T 97a450582be947a48b4aa789389e9e5a P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:08.758059 6172 ts_tablet_manager.cc:1398] T 97a450582be947a48b4aa789389e9e5a P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20240502 21:27:08.763064 31469 consensus_peers.cc:485] T 58be3e32928043e1af387efd99e43b63 P a44f41b0b6644fb6aa29476aed8c964f -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:27:08.764565 31730 consensus_peers.cc:485] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 08d7a0680d244b55b93d17f9ab57b84c -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:27:08.766299 6173 tablet_bootstrap.cc:492] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:27:08.766456 6173 ts_tablet_manager.cc:1367] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.080s user 0.007s sys 0.010s
I20240502 21:27:08.767112 6173 raft_consensus.cc:348] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:08.767290 6173 raft_consensus.cc:374] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:08.767395 6173 raft_consensus.cc:729] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
W20240502 21:27:08.769081 2030 consensus_peers.cc:485] T 565f30d5539c4b7ea956dc49d0f9a55e P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:08.775377 6173 consensus_queue.cc:260] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:08.777231 6173 ts_tablet_manager.cc:1398] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
W20240502 21:27:08.792788 31860 consensus_peers.cc:485] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:27:08.809585 31600 consensus_peers.cc:485] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:27:08.827087 5896 consensus_queue.cc:1035] T e2ed57eb07384dc8953715c2c71b78b8 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:08.827778 31469 consensus_peers.cc:485] T e4e50c50be16429faef86d47cc324bdb P a44f41b0b6644fb6aa29476aed8c964f -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:27:08.835827 31469 consensus_peers.cc:485] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:27:08.842000 6120 consensus_queue.cc:1035] T e2ed57eb07384dc8953715c2c71b78b8 P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:08.844893 2224 raft_consensus.cc:1261] T 565f30d5539c4b7ea956dc49d0f9a55e P d1406e92aec84ac585304e57bd37a40d [term 2 FOLLOWER]: Refusing update from remote peer 1ade0354c44c4c37ae480c7a94b11e1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:27:08.847609 6117 consensus_queue.cc:1035] T 565f30d5539c4b7ea956dc49d0f9a55e P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:27:08.890132 31730 consensus_peers.cc:485] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:27:08.909544 31860 consensus_peers.cc:485] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:27:08.911897 2030 consensus_peers.cc:485] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:27:08.938567 31730 consensus_peers.cc:485] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:27:08.941526 5945 raft_consensus.cc:482] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:08.941676 5945 raft_consensus.cc:504] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:08.942173 5945 leader_election.cc:288] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:27:08.943043 1831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a450582be947a48b4aa789389e9e5a" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:27:08.943254 1831 raft_consensus.cc:2454] T 97a450582be947a48b4aa789389e9e5a P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 0.
I20240502 21:27:08.943722 2032 leader_election.cc:302] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 1e04157b98dd4fad98858c290b73c3db; no voters:
W20240502 21:27:08.943905 2030 leader_election.cc:334] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:08.944175 5945 raft_consensus.cc:2781] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:08.944244 5945 raft_consensus.cc:482] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:08.944332 5945 raft_consensus.cc:3037] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:08.946146 5945 raft_consensus.cc:504] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:08.946626 5945 leader_election.cc:288] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 election: Requested vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:27:08.947180 1831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a450582be947a48b4aa789389e9e5a" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:27:08.947360 1831 raft_consensus.cc:3037] T 97a450582be947a48b4aa789389e9e5a P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:27:08.947763 2030 leader_election.cc:334] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:08.949288 1831 raft_consensus.cc:2454] T 97a450582be947a48b4aa789389e9e5a P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 1.
I20240502 21:27:08.949671 2032 leader_election.cc:302] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, 1e04157b98dd4fad98858c290b73c3db; no voters: 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:08.949887 5945 raft_consensus.cc:2781] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:08.950129 5945 raft_consensus.cc:686] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEADER]: Becoming Leader. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Running, Role: LEADER
I20240502 21:27:08.950420 5945 consensus_queue.cc:237] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:08.954016 31394 catalog_manager.cc:5484] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135). New cstate: current_term: 1 leader_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:08.971210 6004 consensus_queue.cc:1035] T 8702f1a38b274f08affe558f6ef62d74 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:08.975999 6100 consensus_queue.cc:1035] T 8702f1a38b274f08affe558f6ef62d74 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:09.034682 2030 consensus_peers.cc:485] T 939c34669f6447f9a2a88da769d0f84f P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:27:09.042369 31469 consensus_peers.cc:485] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:27:09.047551 31600 consensus_peers.cc:485] T 925797977edf495ca09aa25988d14fac P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:27:09.098620 31600 consensus_peers.cc:485] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:27:09.098981 31394 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:09.105746 1831 raft_consensus.cc:1261] T 97a450582be947a48b4aa789389e9e5a P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer 1ade0354c44c4c37ae480c7a94b11e1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:09.107125 6118 consensus_queue.cc:1035] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:09.110925 2030 consensus_peers.cc:485] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:09.115134 31394 catalog_manager.cc:5576] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863) does not have the latest schema for tablet 97a450582be947a48b4aa789389e9e5a (table rebalancer_dynamic_table [id=2dcceb9804094dee8643679132a7ba30]). Expected version 1 got 0
I20240502 21:27:09.121304 6191 tablet.cc:1722] T 97a450582be947a48b4aa789389e9e5a P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:09.121987 6191 tablet.cc:1620] T 97a450582be947a48b4aa789389e9e5a P 1e04157b98dd4fad98858c290b73c3db: MemRowSet was empty: no flush needed.
I20240502 21:27:09.122431 31394 catalog_manager.cc:5576] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569) does not have the latest schema for tablet 97a450582be947a48b4aa789389e9e5a (table rebalancer_dynamic_table [id=2dcceb9804094dee8643679132a7ba30]). Expected version 1 got 0
I20240502 21:27:09.125769 31388 catalog_manager.cc:5576] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863) does not have the latest schema for tablet 97a450582be947a48b4aa789389e9e5a (table rebalancer_dynamic_table [id=2dcceb9804094dee8643679132a7ba30]). Expected version 1 got 0
W20240502 21:27:09.129248 2030 consensus_peers.cc:485] T b1d59ebecac34cf7b33424cc73cff030 P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:27:09.131290 6190 tablet.cc:1722] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:09.132822 31388 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=2dcceb9804094dee8643679132a7ba30] alter complete (version 1)
I20240502 21:27:09.132786 31394 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_extra_table_50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:09.135089 6190 tablet.cc:1620] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b: MemRowSet was empty: no flush needed.
W20240502 21:27:09.136323 2161 consensus_peers.cc:485] T a8328088928f4de19f7cf20189382d91 P d1406e92aec84ac585304e57bd37a40d -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:27:09.144587 31775 tablet_service.cc:1467] Processing CreateTablet for tablet 056db9c270e94ba58d356f7eaff5c124 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=e8290877eb6949ecaee50d396a2c7a07]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:09.144984 31775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 056db9c270e94ba58d356f7eaff5c124. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:09.144910 31514 tablet_service.cc:1467] Processing CreateTablet for tablet 056db9c270e94ba58d356f7eaff5c124 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=e8290877eb6949ecaee50d396a2c7a07]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:09.145324 31514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 056db9c270e94ba58d356f7eaff5c124. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:09.149317 2076 tablet_service.cc:1467] Processing CreateTablet for tablet 92565312f28a4e3fb0c20a0d0b03f24f (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=e8290877eb6949ecaee50d396a2c7a07]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:09.149327 2077 tablet_service.cc:1467] Processing CreateTablet for tablet 056db9c270e94ba58d356f7eaff5c124 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=e8290877eb6949ecaee50d396a2c7a07]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:09.149732 2076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92565312f28a4e3fb0c20a0d0b03f24f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:09.150020 2077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 056db9c270e94ba58d356f7eaff5c124. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:09.150094 31774 tablet_service.cc:1467] Processing CreateTablet for tablet 54b1ff0b7b574c05b15ff6e8aa9a3c54 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=e8290877eb6949ecaee50d396a2c7a07]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:09.150465 31774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54b1ff0b7b574c05b15ff6e8aa9a3c54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:09.152822 1814 tablet_service.cc:1467] Processing CreateTablet for tablet 54b1ff0b7b574c05b15ff6e8aa9a3c54 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=e8290877eb6949ecaee50d396a2c7a07]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:09.153275 1814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54b1ff0b7b574c05b15ff6e8aa9a3c54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:09.155836 6173 tablet_bootstrap.cc:492] T 92565312f28a4e3fb0c20a0d0b03f24f P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:27:09.156294 31647 tablet_service.cc:1467] Processing CreateTablet for tablet 92565312f28a4e3fb0c20a0d0b03f24f (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=e8290877eb6949ecaee50d396a2c7a07]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:09.156635 31647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92565312f28a4e3fb0c20a0d0b03f24f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:09.156690 1813 tablet_service.cc:1467] Processing CreateTablet for tablet 92565312f28a4e3fb0c20a0d0b03f24f (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=e8290877eb6949ecaee50d396a2c7a07]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:09.157060 1813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92565312f28a4e3fb0c20a0d0b03f24f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:09.157415 6173 tablet_bootstrap.cc:654] T 92565312f28a4e3fb0c20a0d0b03f24f P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:09.158504 31645 tablet_service.cc:1467] Processing CreateTablet for tablet 54b1ff0b7b574c05b15ff6e8aa9a3c54 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=e8290877eb6949ecaee50d396a2c7a07]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:09.158807 31645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54b1ff0b7b574c05b15ff6e8aa9a3c54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:09.180603 6172 tablet_bootstrap.cc:492] T 92565312f28a4e3fb0c20a0d0b03f24f P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:27:09.180728 6194 tablet_bootstrap.cc:492] T 92565312f28a4e3fb0c20a0d0b03f24f P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:27:09.181087 6195 tablet_bootstrap.cc:492] T 056db9c270e94ba58d356f7eaff5c124 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:27:09.182090 6194 tablet_bootstrap.cc:654] T 92565312f28a4e3fb0c20a0d0b03f24f P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:09.182250 6172 tablet_bootstrap.cc:654] T 92565312f28a4e3fb0c20a0d0b03f24f P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:09.182336 6195 tablet_bootstrap.cc:654] T 056db9c270e94ba58d356f7eaff5c124 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:09.184242 6195 tablet_bootstrap.cc:492] T 056db9c270e94ba58d356f7eaff5c124 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:27:09.184428 6195 ts_tablet_manager.cc:1367] T 056db9c270e94ba58d356f7eaff5c124 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 21:27:09.185331 6195 raft_consensus.cc:348] T 056db9c270e94ba58d356f7eaff5c124 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:09.185904 6195 raft_consensus.cc:374] T 056db9c270e94ba58d356f7eaff5c124 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:09.186018 6195 raft_consensus.cc:729] T 056db9c270e94ba58d356f7eaff5c124 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:27:09.186303 6195 consensus_queue.cc:260] T 056db9c270e94ba58d356f7eaff5c124 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
W20240502 21:27:09.186659 2161 consensus_peers.cc:485] T 84ba90ce8dd24ea2b325b1dba2e4c494 P d1406e92aec84ac585304e57bd37a40d -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:27:09.186714 6194 tablet_bootstrap.cc:492] T 92565312f28a4e3fb0c20a0d0b03f24f P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:27:09.186820 6194 ts_tablet_manager.cc:1367] T 92565312f28a4e3fb0c20a0d0b03f24f P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20240502 21:27:09.187009 6195 ts_tablet_manager.cc:1398] T 056db9c270e94ba58d356f7eaff5c124 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:09.187427 6194 raft_consensus.cc:348] T 92565312f28a4e3fb0c20a0d0b03f24f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:09.187588 6194 raft_consensus.cc:374] T 92565312f28a4e3fb0c20a0d0b03f24f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:09.187629 6194 raft_consensus.cc:729] T 92565312f28a4e3fb0c20a0d0b03f24f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:27:09.187788 6194 consensus_queue.cc:260] T 92565312f28a4e3fb0c20a0d0b03f24f P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:09.188230 6194 ts_tablet_manager.cc:1398] T 92565312f28a4e3fb0c20a0d0b03f24f P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:27:09.188515 6194 tablet_bootstrap.cc:492] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:27:09.189894 6194 tablet_bootstrap.cc:654] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:09.209815 6192 tablet_bootstrap.cc:492] T 056db9c270e94ba58d356f7eaff5c124 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
W20240502 21:27:09.213423 31600 consensus_peers.cc:462] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:09.213896 6194 tablet_bootstrap.cc:492] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:27:09.214015 6194 ts_tablet_manager.cc:1367] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.001s
I20240502 21:27:09.214690 6194 raft_consensus.cc:348] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:09.214859 6194 raft_consensus.cc:374] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:09.214910 6194 raft_consensus.cc:729] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:27:09.215108 6194 consensus_queue.cc:260] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:09.215665 6194 ts_tablet_manager.cc:1398] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:27:09.221730 6172 tablet_bootstrap.cc:492] T 92565312f28a4e3fb0c20a0d0b03f24f P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:27:09.222045 6172 ts_tablet_manager.cc:1367] T 92565312f28a4e3fb0c20a0d0b03f24f P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.004s
I20240502 21:27:09.222831 6172 raft_consensus.cc:348] T 92565312f28a4e3fb0c20a0d0b03f24f P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:09.223186 6172 raft_consensus.cc:374] T 92565312f28a4e3fb0c20a0d0b03f24f P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:09.223335 6172 raft_consensus.cc:729] T 92565312f28a4e3fb0c20a0d0b03f24f P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:27:09.223656 6172 consensus_queue.cc:260] T 92565312f28a4e3fb0c20a0d0b03f24f P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:09.224695 6172 ts_tablet_manager.cc:1398] T 92565312f28a4e3fb0c20a0d0b03f24f P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:27:09.225551 6172 tablet_bootstrap.cc:492] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:27:09.227690 6173 tablet_bootstrap.cc:492] T 92565312f28a4e3fb0c20a0d0b03f24f P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:27:09.227870 6173 ts_tablet_manager.cc:1367] T 92565312f28a4e3fb0c20a0d0b03f24f P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.072s user 0.011s sys 0.002s
I20240502 21:27:09.228690 6173 raft_consensus.cc:348] T 92565312f28a4e3fb0c20a0d0b03f24f P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:09.229164 6173 raft_consensus.cc:374] T 92565312f28a4e3fb0c20a0d0b03f24f P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:09.229311 6173 raft_consensus.cc:729] T 92565312f28a4e3fb0c20a0d0b03f24f P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:27:09.229640 6173 consensus_queue.cc:260] T 92565312f28a4e3fb0c20a0d0b03f24f P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:09.229933 6172 tablet_bootstrap.cc:654] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:09.231633 6173 ts_tablet_manager.cc:1398] T 92565312f28a4e3fb0c20a0d0b03f24f P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:09.231940 6173 tablet_bootstrap.cc:492] T 056db9c270e94ba58d356f7eaff5c124 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:27:09.235029 6192 tablet_bootstrap.cc:654] T 056db9c270e94ba58d356f7eaff5c124 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:09.236788 6173 tablet_bootstrap.cc:654] T 056db9c270e94ba58d356f7eaff5c124 P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:09.237437 31388 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:09.250944 6173 tablet_bootstrap.cc:492] T 056db9c270e94ba58d356f7eaff5c124 P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:27:09.251153 6173 ts_tablet_manager.cc:1367] T 056db9c270e94ba58d356f7eaff5c124 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.003s
I20240502 21:27:09.251955 6173 raft_consensus.cc:348] T 056db9c270e94ba58d356f7eaff5c124 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:09.252552 6190 tablet.cc:1722] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:09.252995 6190 tablet.cc:1620] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b: MemRowSet was empty: no flush needed.
I20240502 21:27:09.257150 6191 tablet.cc:1722] T 97a450582be947a48b4aa789389e9e5a P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:09.257560 6191 tablet.cc:1620] T 97a450582be947a48b4aa789389e9e5a P 1e04157b98dd4fad98858c290b73c3db: MemRowSet was empty: no flush needed.
I20240502 21:27:09.252425 6173 raft_consensus.cc:374] T 056db9c270e94ba58d356f7eaff5c124 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:09.260926 6173 raft_consensus.cc:729] T 056db9c270e94ba58d356f7eaff5c124 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:27:09.265141 6172 tablet_bootstrap.cc:492] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:27:09.265653 6172 ts_tablet_manager.cc:1367] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.040s user 0.001s sys 0.010s
I20240502 21:27:09.266939 6172 raft_consensus.cc:348] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:09.267167 6172 raft_consensus.cc:374] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:09.267273 6172 raft_consensus.cc:729] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:27:09.267499 6172 consensus_queue.cc:260] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:09.268493 6172 ts_tablet_manager.cc:1398] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 21:27:09.269099 31394 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=2dcceb9804094dee8643679132a7ba30] alter complete (version 2)
I20240502 21:27:09.275621 6192 tablet_bootstrap.cc:492] T 056db9c270e94ba58d356f7eaff5c124 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:27:09.275823 6192 ts_tablet_manager.cc:1367] T 056db9c270e94ba58d356f7eaff5c124 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.066s user 0.002s sys 0.005s
I20240502 21:27:09.279924 6192 raft_consensus.cc:348] T 056db9c270e94ba58d356f7eaff5c124 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:09.280261 6192 raft_consensus.cc:374] T 056db9c270e94ba58d356f7eaff5c124 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:09.280365 6192 raft_consensus.cc:729] T 056db9c270e94ba58d356f7eaff5c124 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:27:09.280637 6192 consensus_queue.cc:260] T 056db9c270e94ba58d356f7eaff5c124 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:09.281761 6192 ts_tablet_manager.cc:1398] T 056db9c270e94ba58d356f7eaff5c124 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 21:27:09.283200 6192 tablet_bootstrap.cc:492] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
W20240502 21:27:09.283219 31469 consensus_peers.cc:485] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:27:09.261225 6173 consensus_queue.cc:260] T 056db9c270e94ba58d356f7eaff5c124 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:09.284868 6192 tablet_bootstrap.cc:654] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:09.286689 6173 ts_tablet_manager.cc:1398] T 056db9c270e94ba58d356f7eaff5c124 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.035s user 0.002s sys 0.000s
I20240502 21:27:09.298491 6192 tablet_bootstrap.cc:492] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:27:09.298736 6192 ts_tablet_manager.cc:1367] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20240502 21:27:09.299458 6192 raft_consensus.cc:348] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:09.299669 6192 raft_consensus.cc:374] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:09.299758 6192 raft_consensus.cc:729] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:27:09.299983 6192 consensus_queue.cc:260] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:09.300524 6192 ts_tablet_manager.cc:1398] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:09.308912 5705 raft_consensus.cc:482] T 056db9c270e94ba58d356f7eaff5c124 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:09.309062 5705 raft_consensus.cc:504] T 056db9c270e94ba58d356f7eaff5c124 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:09.309734 5705 leader_election.cc:288] T 056db9c270e94ba58d356f7eaff5c124 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:27:09.310084 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "056db9c270e94ba58d356f7eaff5c124" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:27:09.310225 2092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "056db9c270e94ba58d356f7eaff5c124" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:27:09.310292 31797 raft_consensus.cc:2454] T 056db9c270e94ba58d356f7eaff5c124 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:27:09.310451 2092 raft_consensus.cc:2454] T 056db9c270e94ba58d356f7eaff5c124 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:27:09.310720 31471 leader_election.cc:302] T 056db9c270e94ba58d356f7eaff5c124 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:27:09.310966 5705 raft_consensus.cc:2781] T 056db9c270e94ba58d356f7eaff5c124 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:09.311041 5705 raft_consensus.cc:482] T 056db9c270e94ba58d356f7eaff5c124 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:09.311102 5705 raft_consensus.cc:3037] T 056db9c270e94ba58d356f7eaff5c124 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:09.313679 5705 raft_consensus.cc:504] T 056db9c270e94ba58d356f7eaff5c124 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:09.314543 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "056db9c270e94ba58d356f7eaff5c124" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:27:09.314711 31797 raft_consensus.cc:3037] T 056db9c270e94ba58d356f7eaff5c124 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:09.314811 2092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "056db9c270e94ba58d356f7eaff5c124" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:27:09.315037 2092 raft_consensus.cc:3037] T 056db9c270e94ba58d356f7eaff5c124 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1203: Failure
Value of: s.ok()
Actual: false
Expected: true
Runtime error: /tmp/dist-test-taskD89o3N/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 44
Average Replica Count | 21.375000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 2
Maximum | 3
Average | 2.947368
stderr: W20240502 21:26:42.549055 2283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:26:42.549376 2283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:26:42.556699 2283 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:26:44.193477 2283 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 21:26:45.051908 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 408b4cadaf474250a5666b6a49f4f8cc: could not find any suitable replica to move from server 1ade0354c44c4c37ae480c7a94b11e1b to server 08d7a0680d244b55b93d17f9ab57b84c
W20240502 21:26:45.052124 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2e8e8260e3b64486b5712ad7289d1cc8: could not find any suitable replica to move from server a44f41b0b6644fb6aa29476aed8c964f to server 1ade0354c44c4c37ae480c7a94b11e1b
W20240502 21:26:45.052425 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2ef1b694f2424f519337e0d5c334e4d5: could not find any suitable replica to move from server a44f41b0b6644fb6aa29476aed8c964f to server d1406e92aec84ac585304e57bd37a40d
W20240502 21:26:45.052567 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7ff12e5a4c674a5fb7e07cb593390fd1: could not find any suitable replica to move from server 2d50f33557944bb480ea32199d9ee4f1 to server 1e04157b98dd4fad98858c290b73c3db
W20240502 21:26:45.052688 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 654ed3d4f1ef4492ac274c10f2ba1f3d: could not find any suitable replica to move from server a44f41b0b6644fb6aa29476aed8c964f to server 1ade0354c44c4c37ae480c7a94b11e1b
W20240502 21:26:45.052837 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f3134ee4e058445b955a09231a5e159f: could not find any suitable replica to move from server 2d50f33557944bb480ea32199d9ee4f1 to server 1ade0354c44c4c37ae480c7a94b11e1b
W20240502 21:26:45.052974 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff1e1957a03340f382098859ea2c90a8: could not find any suitable replica to move from server 05f4fac41c4f4959b9b316cb112d0ada to server 1ade0354c44c4c37ae480c7a94b11e1b
W20240502 21:26:45.053102 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c7511c25586c4ba086fcfffa353da437: could not find any suitable replica to move from server 08d7a0680d244b55b93d17f9ab57b84c to server 1e04157b98dd4fad98858c290b73c3db
W20240502 21:26:45.053237 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 08ebfd17dee84efd8a264a88ead3fa46: could not find any suitable replica to move from server 05f4fac41c4f4959b9b316cb112d0ada to server 9dec645b473f40578e726f3e50e219bf
W20240502 21:26:45.053422 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5eee55eb24e94c988b4ddbfe2f8442d3: could not find any suitable replica to move from server a44f41b0b6644fb6aa29476aed8c964f to server d1406e92aec84ac585304e57bd37a40d
W20240502 21:26:45.053529 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9e294586148643e080e91357bcbffb17: could not find any suitable replica to move from server 05f4fac41c4f4959b9b316cb112d0ada to server d1406e92aec84ac585304e57bd37a40d
W20240502 21:26:45.053632 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c880d77c61494966b13524216c035e27: could not find any suitable replica to move from server a44f41b0b6644fb6aa29476aed8c964f to server 9dec645b473f40578e726f3e50e219bf
W20240502 21:26:45.053699 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1fffea26d24e40e88ee6944f57c34b08: could not find any suitable replica to move from server 2d50f33557944bb480ea32199d9ee4f1 to server 9dec645b473f40578e726f3e50e219bf
W20240502 21:26:45.053791 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9e962597ef7c41f493f1a41cd94c913b: could not find any suitable replica to move from server 1e04157b98dd4fad98858c290b73c3db to server 05f4fac41c4f4959b9b316cb112d0ada
W20240502 21:26:45.053942 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 59305f5894b34d0085a6cada5a676ca4: could not find any suitable replica to move from server a44f41b0b6644fb6aa29476aed8c964f to server d1406e92aec84ac585304e57bd37a40d
W20240502 21:26:45.054090 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 654ed3d4f1ef4492ac274c10f2ba1f3d: could not find any suitable replica to move from server 05f4fac41c4f4959b9b316cb112d0ada to server 9dec645b473f40578e726f3e50e219bf
W20240502 21:26:45.054184 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c46054f8169d4487b3c912c0681b1c3a: could not find any suitable replica to move from server 05f4fac41c4f4959b9b316cb112d0ada to server 1e04157b98dd4fad98858c290b73c3db
W20240502 21:26:45.054293 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0af22f3605164ff58dc3914737c2de4f: could not find any suitable replica to move from server 2d50f33557944bb480ea32199d9ee4f1 to server 1ade0354c44c4c37ae480c7a94b11e1b
W20240502 21:26:45.054419 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ba6507441fb04f29a519bb81d3dba0e1: could not find any suitable replica to move from server 08d7a0680d244b55b93d17f9ab57b84c to server 1e04157b98dd4fad98858c290b73c3db
W20240502 21:26:45.054490 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff1e1957a03340f382098859ea2c90a8: could not find any suitable replica to move from server 2d50f33557944bb480ea32199d9ee4f1 to server 9dec645b473f40578e726f3e50e219bf
W20240502 21:26:45.054546 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c7511c25586c4ba086fcfffa353da437: could not find any suitable replica to move from server a44f41b0b6644fb6aa29476aed8c964f to server 1e04157b98dd4fad98858c290b73c3db
W20240502 21:26:45.054633 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 08ebfd17dee84efd8a264a88ead3fa46: could not find any suitable replica to move from server 2d50f33557944bb480ea32199d9ee4f1 to server 1ade0354c44c4c37ae480c7a94b11e1b
W20240502 21:26:45.054729 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9e294586148643e080e91357bcbffb17: could not find any suitable replica to move from server 08d7a0680d244b55b93d17f9ab57b84c to server d1406e92aec84ac585304e57bd37a40d
I20240502 21:26:45.090737 2283 rebalancer_tool.cc:1074] tablet bb64e0d78397429d959535f570658594: '05f4fac41c4f4959b9b316cb112d0ada' -> '9dec645b473f40578e726f3e50e219bf' move scheduled
I20240502 21:26:45.142215 2283 rebalancer_tool.cc:1074] tablet 182b3f15c06c4c07ade0a8854d3ecafd: '08d7a0680d244b55b93d17f9ab57b84c' -> '1e04157b98dd4fad98858c290b73c3db' move scheduled
I20240502 21:26:45.189092 2283 rebalancer_tool.cc:1074] tablet f02f83ec0a494da5b66241d9b379a03f: 'd1406e92aec84ac585304e57bd37a40d' -> 'a44f41b0b6644fb6aa29476aed8c964f' move scheduled
I20240502 21:26:45.245437 2283 rebalancer_tool.cc:1074] tablet 49acc5301b9849e9b6795179acbc59a8: '1ade0354c44c4c37ae480c7a94b11e1b' -> '2d50f33557944bb480ea32199d9ee4f1' move scheduled
I20240502 21:26:45.248075 2283 rebalancer_tool.cc:1084] tablet 9a0d1c43a0794c2884dfef498f046f6d: '05f4fac41c4f4959b9b316cb112d0ada' -> '9dec645b473f40578e726f3e50e219bf' move ignored: Not found: 9a0d1c43a0794c2884dfef498f046f6d: tablet not found
I20240502 21:26:45.248219 2283 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 21:26:46.144237 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5fea5cf0a15d489c9c9a523859a65476: could not find any suitable replica to move from server 1e04157b98dd4fad98858c290b73c3db to server 05f4fac41c4f4959b9b316cb112d0ada
W20240502 21:26:46.144425 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 654ed3d4f1ef4492ac274c10f2ba1f3d: could not find any suitable replica to move from server a44f41b0b6644fb6aa29476aed8c964f to server d1406e92aec84ac585304e57bd37a40d
W20240502 21:26:46.144596 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ba6507441fb04f29a519bb81d3dba0e1: could not find any suitable replica to move from server 2d50f33557944bb480ea32199d9ee4f1 to server 1e04157b98dd4fad98858c290b73c3db
W20240502 21:26:46.144726 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1fffea26d24e40e88ee6944f57c34b08: could not find any suitable replica to move from server 2d50f33557944bb480ea32199d9ee4f1 to server 1e04157b98dd4fad98858c290b73c3db
W20240502 21:26:46.144883 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9e294586148643e080e91357bcbffb17: could not find any suitable replica to move from server 2d50f33557944bb480ea32199d9ee4f1 to server 1ade0354c44c4c37ae480c7a94b11e1b
W20240502 21:26:46.144995 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9e962597ef7c41f493f1a41cd94c913b: could not find any suitable replica to move from server a44f41b0b6644fb6aa29476aed8c964f to server 9dec645b473f40578e726f3e50e219bf
W20240502 21:26:46.145104 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 59305f5894b34d0085a6cada5a676ca4: could not find any suitable replica to move from server 08d7a0680d244b55b93d17f9ab57b84c to server 1ade0354c44c4c37ae480c7a94b11e1b
W20240502 21:26:46.145267 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5fea5cf0a15d489c9c9a523859a65476: could not find any suitable replica to move from server d1406e92aec84ac585304e57bd37a40d to server a44f41b0b6644fb6aa29476aed8c964f
W20240502 21:26:46.145490 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7ff12e5a4c674a5fb7e07cb593390fd1: could not find any suitable replica to move from server 2d50f33557944bb480ea32199d9ee4f1 to server 1e04157b98dd4fad98858c290b73c3db
W20240502 21:26:46.145603 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 08ebfd17dee84efd8a264a88ead3fa46: could not find any suitable replica to move from server 05f4fac41c4f4959b9b316cb112d0ada to server 1ade0354c44c4c37ae480c7a94b11e1b
W20240502 21:26:46.145725 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5eee55eb24e94c988b4ddbfe2f8442d3: could not find any suitable replica to move from server a44f41b0b6644fb6aa29476aed8c964f to server 1e04157b98dd4fad98858c290b73c3db
W20240502 21:26:46.145808 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c7511c25586c4ba086fcfffa353da437: could not find any suitable replica to move from server 2d50f33557944bb480ea32199d9ee4f1 to server 1e04157b98dd4fad98858c290b73c3db
W20240502 21:26:46.145892 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2e8e8260e3b64486b5712ad7289d1cc8: could not find any suitable replica to move from server 2d50f33557944bb480ea32199d9ee4f1 to server 1e04157b98dd4fad98858c290b73c3db
W20240502 21:26:46.145975 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9e294586148643e080e91357bcbffb17: could not find any suitable replica to move from server 08d7a0680d244b55b93d17f9ab57b84c to server d1406e92aec84ac585304e57bd37a40d
W20240502 21:26:46.146060 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c880d77c61494966b13524216c035e27: could not find any suitable replica to move from server a44f41b0b6644fb6aa29476aed8c964f to server 1ade0354c44c4c37ae480c7a94b11e1b
W20240502 21:26:46.146227 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2ef1b694f2424f519337e0d5c334e4d5: could not find any suitable replica to move from server a44f41b0b6644fb6aa29476aed8c964f to server 1ade0354c44c4c37ae480c7a94b11e1b
W20240502 21:26:46.146342 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c46054f8169d4487b3c912c0681b1c3a: could not find any suitable replica to move from server 05f4fac41c4f4959b9b316cb112d0ada to server 1ade0354c44c4c37ae480c7a94b11e1b
W20240502 21:26:46.146430 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0af22f3605164ff58dc3914737c2de4f: could not find any suitable replica to move from server 05f4fac41c4f4959b9b316cb112d0ada to server 1ade0354c44c4c37ae480c7a94b11e1b
W20240502 21:26:46.146656 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 08ebfd17dee84efd8a264a88ead3fa46: could not find any suitable replica to move from server 2d50f33557944bb480ea32199d9ee4f1 to server 9dec645b473f40578e726f3e50e219bf
W20240502 21:26:46.146757 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1fffea26d24e40e88ee6944f57c34b08: could not find any suitable replica to move from server 05f4fac41c4f4959b9b316cb112d0ada to server 1e04157b98dd4fad98858c290b73c3db
W20240502 21:26:46.146857 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c7511c25586c4ba086fcfffa353da437: could not find any suitable replica to move from server 08d7a0680d244b55b93d17f9ab57b84c to server d1406e92aec84ac585304e57bd37a40d
W20240502 21:26:46.146947 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2e8e8260e3b64486b5712ad7289d1cc8: could not find any suitable replica to move from server a44f41b0b6644fb6aa29476aed8c964f to server 9dec645b473f40578e726f3e50e219bf
I20240502 21:26:46.220538 2283 rebalancer_tool.cc:1074] tablet b407c2c9b3a14ce3ada52b7856c6ecee: '08d7a0680d244b55b93d17f9ab57b84c' -> '9dec645b473f40578e726f3e50e219bf' move scheduled
I20240502 21:26:46.313071 2283 rebalancer_tool.cc:1074] tablet fe4e41225da24d7c91c6bf7733a5e067: '05f4fac41c4f4959b9b316cb112d0ada' -> '1ade0354c44c4c37ae480c7a94b11e1b' move scheduled
I20240502 21:26:46.416746 2283 rebalancer_tool.cc:1074] tablet 62f083e9641042c990efcfb1c4104de5: 'a44f41b0b6644fb6aa29476aed8c964f' -> 'd1406e92aec84ac585304e57bd37a40d' move scheduled
I20240502 21:26:46.448814 2283 rebalancer_tool.cc:1074] tablet 815b1fc784714c8cb4ba76707c345625: '1e04157b98dd4fad98858c290b73c3db' -> '2d50f33557944bb480ea32199d9ee4f1' move scheduled
I20240502 21:26:46.476989 2283 rebalancer_tool.cc:1074] tablet 55fc18e4cc98446ca50c1dd2b6a9a89d: '08d7a0680d244b55b93d17f9ab57b84c' -> '9dec645b473f40578e726f3e50e219bf' move scheduled
I20240502 21:26:46.539757 2283 rebalancer_tool.cc:1074] tablet b9342dbe0497499ba4c238f2ed033293: '1ade0354c44c4c37ae480c7a94b11e1b' -> '05f4fac41c4f4959b9b316cb112d0ada' move scheduled
I20240502 21:26:46.595197 2283 rebalancer_tool.cc:1074] tablet 5b1dcae186f542cea92c9191fa8ef951: 'a44f41b0b6644fb6aa29476aed8c964f' -> 'd1406e92aec84ac585304e57bd37a40d' move scheduled
I20240502 21:26:46.634104 2283 rebalancer_tool.cc:1074] tablet 9ea8ddb9f8e04893b56c10dbbb5af321: '2d50f33557944bb480ea32199d9ee4f1' -> '1e04157b98dd4fad98858c290b73c3db' move scheduled
I20240502 21:26:46.743669 2283 rebalancer_tool.cc:1074] tablet f9bbc56ba2f841b9bb36a864e1a1f808: '08d7a0680d244b55b93d17f9ab57b84c' -> '9dec645b473f40578e726f3e50e219bf' move scheduled
I20240502 21:26:47.118072 2283 rebalancer_tool.cc:1074] tablet 83904952a5a541ecac508449e23e70dd: 'a44f41b0b6644fb6aa29476aed8c964f' -> '1ade0354c44c4c37ae480c7a94b11e1b' move scheduled
I20240502 21:26:47.120296 2283 rebalancer_tool.cc:1084] tablet 80023f3b254c4369954ab92a085a0ec5: '05f4fac41c4f4959b9b316cb112d0ada' -> 'd1406e92aec84ac585304e57bd37a40d' move ignored: Not found: 80023f3b254c4369954ab92a085a0ec5: tablet not found
I20240502 21:26:47.120405 2283 rebalancer_tool.cc:756] re-synchronizing cluster state
E20240502 21:26:48.089385 3073 ksck_remote.cc:634] unable to open table rebalancer_dynamic_table_new_name: Not found: the table does not exist: table_name: "rebalancer_dynamic_table_new_name"
I20240502 21:26:48.356667 2283 rebalancer_tool.cc:1136] tablet 815b1fc784714c8cb4ba76707c345625: '1e04157b98dd4fad98858c290b73c3db' -> '2d50f33557944bb480ea32199d9ee4f1' move completed: OK
I20240502 21:26:48.396911 2283 rebalancer_tool.cc:1136] tablet 182b3f15c06c4c07ade0a8854d3ecafd: '08d7a0680d244b55b93d17f9ab57b84c' -> '1e04157b98dd4fad98858c290b73c3db' move completed: OK
I20240502 21:26:48.398589 2283 rebalancer_tool.cc:1123] tablet bb64e0d78397429d959535f570658594: '05f4fac41c4f4959b9b316cb112d0ada' -> '9dec645b473f40578e726f3e50e219bf' move is abandoned: Not found: bb64e0d78397429d959535f570658594: tablet not found
I20240502 21:26:48.425632 2283 rebalancer_tool.cc:1136] tablet f02f83ec0a494da5b66241d9b379a03f: 'd1406e92aec84ac585304e57bd37a40d' -> 'a44f41b0b6644fb6aa29476aed8c964f' move completed: OK
I20240502 21:26:48.592027 2283 rebalancer_tool.cc:1136] tablet 49acc5301b9849e9b6795179acbc59a8: '1ade0354c44c4c37ae480c7a94b11e1b' -> '2d50f33557944bb480ea32199d9ee4f1' move completed: OK
I20240502 21:26:48.595192 2283 rebalancer_tool.cc:1123] tablet fe4e41225da24d7c91c6bf7733a5e067: '05f4fac41c4f4959b9b316cb112d0ada' -> '1ade0354c44c4c37ae480c7a94b11e1b' move is abandoned: Not found: fe4e41225da24d7c91c6bf7733a5e067: tablet not found
I20240502 21:26:48.892694 2283 rebalancer_tool.cc:1136] tablet 9ea8ddb9f8e04893b56c10dbbb5af321: '2d50f33557944bb480ea32199d9ee4f1' -> '1e04157b98dd4fad98858c290b73c3db' move completed: OK
I20240502 21:26:49.019140 2283 rebalancer_tool.cc:1136] tablet 5b1dcae186f542cea92c9191fa8ef951: 'a44f41b0b6644fb6aa29476aed8c964f' -> 'd1406e92aec84ac585304e57bd37a40d' move completed: OK
I20240502 21:26:49.113278 2283 rebalancer_tool.cc:1136] tablet b9342dbe0497499ba4c238f2ed033293: '1ade0354c44c4c37ae480c7a94b11e1b' -> '05f4fac41c4f4959b9b316cb112d0ada' move completed: OK
I20240502 21:26:49.152899 2283 rebalancer_tool.cc:1136] tablet 62f083e9641042c990efcfb1c4104de5: 'a44f41b0b6644fb6aa29476aed8c964f' -> 'd1406e92aec84ac585304e57bd37a40d' move completed: OK
I20240502 21:26:49.167753 2283 rebalancer_tool.cc:1136] tablet 55fc18e4cc98446ca50c1dd2b6a9a89d: '08d7a0680d244b55b93d17f9ab57b84c' -> '9dec645b473f40578e726f3e50e219bf' move completed: OK
I20240502 21:26:49.190116 2283 rebalancer_tool.cc:1136] tablet 83904952a5a541ecac508449e23e70dd: 'a44f41b0b6644fb6aa29476aed8c964f' -> '1ade0354c44c4c37ae480c7a94b11e1b' move completed: OK
I20240502 21:26:49.494248 2283 rebalancer_tool.cc:1136] tablet b407c2c9b3a14ce3ada52b7856c6ecee: '08d7a0680d244b55b93d17f9ab57b84c' -> '9dec645b473f40578e726f3e50e219bf' move completed: OK
I20240502 21:26:50.184290 2283 rebalancer_tool.cc:1136] tablet f9bbc56ba2f841b9bb36a864e1a1f808: '08d7a0680d244b55b93d17f9ab57b84c' -> '9dec645b473f40578e726f3e50e219bf' move completed: OK
W20240502 21:26:50.811515 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4cfb4e8589a24f279c0e7f552012212c: could not find any suitable replica to move from server 1e04157b98dd4fad98858c290b73c3db to server a44f41b0b6644fb6aa29476aed8c964f
W20240502 21:26:50.811730 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d8ac6ebe732c415bb64d1aecfeadc621: could not find any suitable replica to move from server 1ade0354c44c4c37ae480c7a94b11e1b to server 05f4fac41c4f4959b9b316cb112d0ada
W20240502 21:26:50.811931 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 112fc5f7af4e4c8080926e0a6128b468: could not find any suitable replica to move from server 2d50f33557944bb480ea32199d9ee4f1 to server 05f4fac41c4f4959b9b316cb112d0ada
W20240502 21:26:50.812009 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2e8e8260e3b64486b5712ad7289d1cc8: could not find any suitable replica to move from server 2d50f33557944bb480ea32199d9ee4f1 to server 1ade0354c44c4c37ae480c7a94b11e1b
W20240502 21:26:50.812274 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 654ed3d4f1ef4492ac274c10f2ba1f3d: could not find any suitable replica to move from server 05f4fac41c4f4959b9b316cb112d0ada to server 9dec645b473f40578e726f3e50e219bf
W20240502 21:26:50.812387 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9e294586148643e080e91357bcbffb17: could not find any suitable replica to move from server 2d50f33557944bb480ea32199d9ee4f1 to server d1406e92aec84ac585304e57bd37a40d
W20240502 21:26:50.812552 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d8ac6ebe732c415bb64d1aecfeadc621: could not find any suitable replica to move from server 9dec645b473f40578e726f3e50e219bf to server 2d50f33557944bb480ea32199d9ee4f1
W20240502 21:26:50.812640 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5eee55eb24e94c988b4ddbfe2f8442d3: could not find any suitable replica to move from server a44f41b0b6644fb6aa29476aed8c964f to server 1e04157b98dd4fad98858c290b73c3db
W20240502 21:26:50.812738 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0af22f3605164ff58dc3914737c2de4f: could not find any suitable replica to move from server a44f41b0b6644fb6aa29476aed8c964f to server 1e04157b98dd4fad98858c290b73c3db
W20240502 21:26:50.812839 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 654ed3d4f1ef4492ac274c10f2ba1f3d: could not find any suitable replica to move from server a44f41b0b6644fb6aa29476aed8c964f to server 1ade0354c44c4c37ae480c7a94b11e1b
W20240502 21:26:50.812954 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 408b4cadaf474250a5666b6a49f4f8cc: could not find any suitable replica to move from server 1ade0354c44c4c37ae480c7a94b11e1b to server 9dec645b473f40578e726f3e50e219bf
I20240502 21:26:50.837576 2283 rebalancer_tool.cc:1074] tablet dab4f06cc78946dc962267cb2aae04f7: 'd1406e92aec84ac585304e57bd37a40d' -> '1ade0354c44c4c37ae480c7a94b11e1b' move scheduled
I20240502 21:26:50.878937 2283 rebalancer_tool.cc:1074] tablet 62f083e9641042c990efcfb1c4104de5: '08d7a0680d244b55b93d17f9ab57b84c' -> '9dec645b473f40578e726f3e50e219bf' move scheduled
I20240502 21:26:50.963346 2283 rebalancer_tool.cc:1074] tablet a1fd14de204c40d98027ecb329ff79f3: '05f4fac41c4f4959b9b316cb112d0ada' -> '1e04157b98dd4fad98858c290b73c3db' move scheduled
I20240502 21:26:50.993579 2283 rebalancer_tool.cc:1074] tablet 53b7eec870c44999b48f0bf744b2ffcd: '2d50f33557944bb480ea32199d9ee4f1' -> 'd1406e92aec84ac585304e57bd37a40d' move scheduled
I20240502 21:26:51.033740 2283 rebalancer_tool.cc:1074] tablet 88189733d9a14560971dbeb71543073f: 'a44f41b0b6644fb6aa29476aed8c964f' -> '1ade0354c44c4c37ae480c7a94b11e1b' move scheduled
I20240502 21:26:51.064757 2283 rebalancer_tool.cc:1074] tablet 759b7529d7d84baab1a4622d34f803bf: '08d7a0680d244b55b93d17f9ab57b84c' -> '9dec645b473f40578e726f3e50e219bf' move scheduled
I20240502 21:26:51.099740 2283 rebalancer_tool.cc:1074] tablet 5a54d2aad63c4f3182992f00b8564acf: '05f4fac41c4f4959b9b316cb112d0ada' -> 'd1406e92aec84ac585304e57bd37a40d' move scheduled
I20240502 21:26:51.147735 2283 rebalancer_tool.cc:1074] tablet 5bc6c7211d5f4f73b2a050b9ec78c6b1: '1e04157b98dd4fad98858c290b73c3db' -> '2d50f33557944bb480ea32199d9ee4f1' move scheduled
I20240502 21:26:51.230374 2283 rebalancer_tool.cc:1074] tablet c5c344f8f84f4a73b853b4f908b93964: 'a44f41b0b6644fb6aa29476aed8c964f' -> '1ade0354c44c4c37ae480c7a94b11e1b' move scheduled
I20240502 21:26:51.292483 2283 rebalancer_tool.cc:1074] tablet ae56d18c629a4e5eb50911c89de4227c: '08d7a0680d244b55b93d17f9ab57b84c' -> '9dec645b473f40578e726f3e50e219bf' move scheduled
I20240502 21:26:51.327617 2283 rebalancer_tool.cc:1074] tablet 55fc18e4cc98446ca50c1dd2b6a9a89d: '05f4fac41c4f4959b9b316cb112d0ada' -> 'd1406e92aec84ac585304e57bd37a40d' move scheduled
I20240502 21:26:51.367913 2283 rebalancer_tool.cc:1074] tablet 5b1dcae186f542cea92c9191fa8ef951: '2d50f33557944bb480ea32199d9ee4f1' -> '1e04157b98dd4fad98858c290b73c3db' move scheduled
I20240502 21:26:51.423676 2283 rebalancer_tool.cc:1074] tablet acb3bdc518b84b5abc362cc4e12027c3: '1ade0354c44c4c37ae480c7a94b11e1b' -> 'a44f41b0b6644fb6aa29476aed8c964f' move scheduled
I20240502 21:26:51.599697 2283 rebalancer_tool.cc:1074] tablet b9342dbe0497499ba4c238f2ed033293: '08d7a0680d244b55b93d17f9ab57b84c' -> '9dec645b473f40578e726f3e50e219bf' move scheduled
I20240502 21:26:51.812785 2283 rebalancer_tool.cc:1074] tablet 8ab82de23912483ea178376a5ba5cdc1: '05f4fac41c4f4959b9b316cb112d0ada' -> 'd1406e92aec84ac585304e57bd37a40d' move scheduled
I20240502 21:26:52.247937 2283 rebalancer_tool.cc:1074] tablet 2db9f537acba4424bca97ffc05c6d0b4: '1e04157b98dd4fad98858c290b73c3db' -> 'a44f41b0b6644fb6aa29476aed8c964f' move scheduled
I20240502 21:26:52.448941 2283 rebalancer_tool.cc:1074] tablet f5e82c0ab5c441efbcfbcd383b8db82f: '1ade0354c44c4c37ae480c7a94b11e1b' -> '08d7a0680d244b55b93d17f9ab57b84c' move scheduled
I20240502 21:26:52.892624 2283 rebalancer_tool.cc:1074] tablet fc93b5adc1d94e809e97e56b56822611: 'a44f41b0b6644fb6aa29476aed8c964f' -> '9dec645b473f40578e726f3e50e219bf' move scheduled
I20240502 21:26:53.622874 2283 rebalancer_tool.cc:1136] tablet acb3bdc518b84b5abc362cc4e12027c3: '1ade0354c44c4c37ae480c7a94b11e1b' -> 'a44f41b0b6644fb6aa29476aed8c964f' move completed: OK
I20240502 21:26:53.886734 2283 rebalancer_tool.cc:1136] tablet 62f083e9641042c990efcfb1c4104de5: '08d7a0680d244b55b93d17f9ab57b84c' -> '9dec645b473f40578e726f3e50e219bf' move completed: OK
I20240502 21:26:54.130336 2283 rebalancer_tool.cc:1136] tablet a1fd14de204c40d98027ecb329ff79f3: '05f4fac41c4f4959b9b316cb112d0ada' -> '1e04157b98dd4fad98858c290b73c3db' move completed: OK
I20240502 21:26:54.152088 2283 rebalancer_tool.cc:1136] tablet ae56d18c629a4e5eb50911c89de4227c: '08d7a0680d244b55b93d17f9ab57b84c' -> '9dec645b473f40578e726f3e50e219bf' move completed: OK
I20240502 21:26:54.254511 2283 rebalancer_tool.cc:1136] tablet b9342dbe0497499ba4c238f2ed033293: '08d7a0680d244b55b93d17f9ab57b84c' -> '9dec645b473f40578e726f3e50e219bf' move completed: OK
I20240502 21:26:54.264850 2283 rebalancer_tool.cc:1136] tablet 8ab82de23912483ea178376a5ba5cdc1: '05f4fac41c4f4959b9b316cb112d0ada' -> 'd1406e92aec84ac585304e57bd37a40d' move completed: OK
I20240502 21:26:54.274919 2283 rebalancer_tool.cc:1136] tablet 88189733d9a14560971dbeb71543073f: 'a44f41b0b6644fb6aa29476aed8c964f' -> '1ade0354c44c4c37ae480c7a94b11e1b' move completed: OK
I20240502 21:26:54.288460 2283 rebalancer_tool.cc:1136] tablet dab4f06cc78946dc962267cb2aae04f7: 'd1406e92aec84ac585304e57bd37a40d' -> '1ade0354c44c4c37ae480c7a94b11e1b' move completed: OK
I20240502 21:26:54.305087 2283 rebalancer_tool.cc:1136] tablet 5bc6c7211d5f4f73b2a050b9ec78c6b1: '1e04157b98dd4fad98858c290b73c3db' -> '2d50f33557944bb480ea32199d9ee4f1' move completed: OK
I20240502 21:26:54.317422 2283 rebalancer_tool.cc:1136] tablet 759b7529d7d84baab1a4622d34f803bf: '08d7a0680d244b55b93d17f9ab57b84c' -> '9dec645b473f40578e726f3e50e219bf' move completed: OK
I20240502 21:26:54.371966 2283 rebalancer_tool.cc:1136] tablet 53b7eec870c44999b48f0bf744b2ffcd: '2d50f33557944bb480ea32199d9ee4f1' -> 'd1406e92aec84ac585304e57bd37a40d' move completed: OK
I20240502 21:26:54.382179 2283 rebalancer_tool.cc:1136] tablet 5a54d2aad63c4f3182992f00b8564acf: '05f4fac41c4f4959b9b316cb112d0ada' -> 'd1406e92aec84ac585304e57bd37a40d' move completed: OK
I20240502 21:26:54.391726 2283 rebalancer_tool.cc:1136] tablet c5c344f8f84f4a73b853b4f908b93964: 'a44f41b0b6644fb6aa29476aed8c964f' -> '1ade0354c44c4c37ae480c7a94b11e1b' move completed: OK
I20240502 21:26:54.401224 2283 rebalancer_tool.cc:1136] tablet 55fc18e4cc98446ca50c1dd2b6a9a89d: '05f4fac41c4f4959b9b316cb112d0ada' -> 'd1406e92aec84ac585304e57bd37a40d' move completed: OK
I20240502 21:26:54.413606 2283 rebalancer_tool.cc:1136] tablet 2db9f537acba4424bca97ffc05c6d0b4: '1e04157b98dd4fad98858c290b73c3db' -> 'a44f41b0b6644fb6aa29476aed8c964f' move completed: OK
I20240502 21:26:54.465866 2283 rebalancer_tool.cc:1074] tablet 46279270b2334633906af57c4b43e969: 'd1406e92aec84ac585304e57bd37a40d' -> '05f4fac41c4f4959b9b316cb112d0ada' move scheduled
I20240502 21:26:54.509835 2283 rebalancer_tool.cc:1074] tablet 0b8d119ee8474a40ac08df61df5b8423: '08d7a0680d244b55b93d17f9ab57b84c' -> '9dec645b473f40578e726f3e50e219bf' move scheduled
I20240502 21:26:54.528662 2283 rebalancer_tool.cc:1074] tablet b20df8896a2b423cb53c2c1a2a1b98e2: '2d50f33557944bb480ea32199d9ee4f1' -> 'd1406e92aec84ac585304e57bd37a40d' move scheduled
I20240502 21:26:54.568661 2283 rebalancer_tool.cc:1074] tablet 1022c4c6ee8642ca95250933a940fe7a: '1ade0354c44c4c37ae480c7a94b11e1b' -> '05f4fac41c4f4959b9b316cb112d0ada' move scheduled
I20240502 21:26:54.614777 2283 rebalancer_tool.cc:1074] tablet de2216a74f8344779a881651b66a85e5: '1e04157b98dd4fad98858c290b73c3db' -> '08d7a0680d244b55b93d17f9ab57b84c' move scheduled
I20240502 21:26:54.738895 2283 rebalancer_tool.cc:1074] tablet 182b3f15c06c4c07ade0a8854d3ecafd: 'a44f41b0b6644fb6aa29476aed8c964f' -> '9dec645b473f40578e726f3e50e219bf' move scheduled
I20240502 21:26:54.788697 2283 rebalancer_tool.cc:1074] tablet 1ed225bdefc64ddd8eef2109bfd969df: '2d50f33557944bb480ea32199d9ee4f1' -> 'd1406e92aec84ac585304e57bd37a40d' move scheduled
I20240502 21:26:54.855793 2283 rebalancer_tool.cc:1074] tablet 577e7a586ad9459f8e77ce4b8f563c09: '1ade0354c44c4c37ae480c7a94b11e1b' -> '05f4fac41c4f4959b9b316cb112d0ada' move scheduled
I20240502 21:26:55.029058 2283 rebalancer_tool.cc:1074] tablet 83904952a5a541ecac508449e23e70dd: '08d7a0680d244b55b93d17f9ab57b84c' -> '1e04157b98dd4fad98858c290b73c3db' move scheduled
I20240502 21:26:55.324613 2283 rebalancer_tool.cc:1074] tablet f13c3b9c4b7c471fa34156b781473ec4: 'd1406e92aec84ac585304e57bd37a40d' -> 'a44f41b0b6644fb6aa29476aed8c964f' move scheduled
I20240502 21:26:55.326303 2283 rebalancer_tool.cc:1084] tablet aca362d1a6894071b195dd9fb7594b1f: '1ade0354c44c4c37ae480c7a94b11e1b' -> '9dec645b473f40578e726f3e50e219bf' move ignored: Not found: aca362d1a6894071b195dd9fb7594b1f: tablet not found
I20240502 21:26:55.326402 2283 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 21:26:56.337721 4345 meta_cache.cc:1041] Time spent processing tablet locations response: real 0.099s user 0.000s sys 0.000s
W20240502 21:26:56.338531 4345 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.24.226.190:43531, user_credentials={real_user=slave}} blocked reactor thread for 99939.2us
W20240502 21:26:57.415488 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4cfb4e8589a24f279c0e7f552012212c: could not find any suitable replica to move from server 1e04157b98dd4fad98858c290b73c3db to server a44f41b0b6644fb6aa29476aed8c964f
W20240502 21:26:57.415987 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5eee55eb24e94c988b4ddbfe2f8442d3: could not find any suitable replica to move from server 08d7a0680d244b55b93d17f9ab57b84c to server 1ade0354c44c4c37ae480c7a94b11e1b
W20240502 21:26:57.416111 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2e8e8260e3b64486b5712ad7289d1cc8: could not find any suitable replica to move from server 2d50f33557944bb480ea32199d9ee4f1 to server 1e04157b98dd4fad98858c290b73c3db
W20240502 21:26:57.416400 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d8ac6ebe732c415bb64d1aecfeadc621: could not find any suitable replica to move from server 9dec645b473f40578e726f3e50e219bf to server 2d50f33557944bb480ea32199d9ee4f1
W20240502 21:26:57.416532 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cc412415870b4447b2e9707c2f7fd323: could not find any suitable replica to move from server d1406e92aec84ac585304e57bd37a40d to server 08d7a0680d244b55b93d17f9ab57b84c
W20240502 21:26:57.416636 2283 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 654ed3d4f1ef4492ac274c10f2ba1f3d: could not find any suitable replica to move from server 05f4fac41c4f4959b9b316cb112d0ada to server d1406e92aec84ac585304e57bd37a40d
I20240502 21:26:57.451498 2283 rebalancer_tool.cc:1074] tablet 2c737c49a32b4f3ea5ef735fa254d48e: '1e04157b98dd4fad98858c290b73c3db' -> '05f4fac41c4f4959b9b316cb112d0ada' move scheduled
I20240502 21:26:57.495048 2283 rebalancer_tool.cc:1074] tablet 2068521c26dc44f9ba60e274eb039b22: '1ade0354c44c4c37ae480c7a94b11e1b' -> 'a44f41b0b6644fb6aa29476aed8c964f' move scheduled
I20240502 21:26:57.565687 2283 rebalancer_tool.cc:1074] tablet 0d923caa3f034a899050fe62dfc0c3d8: '9dec645b473f40578e726f3e50e219bf' -> '08d7a0680d244b55b93d17f9ab57b84c' move scheduled
I20240502 21:26:57.658532 2283 rebalancer_tool.cc:1074] tablet 534b4d96df9b4bc5a9480b6c45eb11a5: 'd1406e92aec84ac585304e57bd37a40d' -> '2d50f33557944bb480ea32199d9ee4f1' move scheduled
I20240502 21:26:57.717654 2283 rebalancer_tool.cc:1074] tablet 53b7eec870c44999b48f0bf744b2ffcd: 'a44f41b0b6644fb6aa29476aed8c964f' -> '1e04157b98dd4fad98858c290b73c3db' move scheduled
I20240502 21:26:57.811379 2283 rebalancer_tool.cc:1074] tablet e0f96b56fb68439a97cf2aae4aa89b57: '05f4fac41c4f4959b9b316cb112d0ada' -> '1ade0354c44c4c37ae480c7a94b11e1b' move scheduled
I20240502 21:26:57.904922 2283 rebalancer_tool.cc:1136] tablet 83904952a5a541ecac508449e23e70dd: '08d7a0680d244b55b93d17f9ab57b84c' -> '1e04157b98dd4fad98858c290b73c3db' move completed: OK
I20240502 21:26:57.912956 2283 rebalancer_tool.cc:1136] tablet 577e7a586ad9459f8e77ce4b8f563c09: '1ade0354c44c4c37ae480c7a94b11e1b' -> '05f4fac41c4f4959b9b316cb112d0ada' move completed: OK
I20240502 21:26:57.921365 2283 rebalancer_tool.cc:1136] tablet 1ed225bdefc64ddd8eef2109bfd969df: '2d50f33557944bb480ea32199d9ee4f1' -> 'd1406e92aec84ac585304e57bd37a40d' move completed: OK
I20240502 21:26:57.940271 2283 rebalancer_tool.cc:1136] tablet 182b3f15c06c4c07ade0a8854d3ecafd: 'a44f41b0b6644fb6aa29476aed8c964f' -> '9dec645b473f40578e726f3e50e219bf' move completed: OK
I20240502 21:26:57.953073 2283 rebalancer_tool.cc:1136] tablet de2216a74f8344779a881651b66a85e5: '1e04157b98dd4fad98858c290b73c3db' -> '08d7a0680d244b55b93d17f9ab57b84c' move completed: OK
I20240502 21:26:58.015239 2283 rebalancer_tool.cc:1136] tablet 1022c4c6ee8642ca95250933a940fe7a: '1ade0354c44c4c37ae480c7a94b11e1b' -> '05f4fac41c4f4959b9b316cb112d0ada' move completed: OK
I20240502 21:26:58.135919 2283 rebalancer_tool.cc:1136] tablet b20df8896a2b423cb53c2c1a2a1b98e2: '2d50f33557944bb480ea32199d9ee4f1' -> 'd1406e92aec84ac585304e57bd37a40d' move completed: OK
I20240502 21:26:58.145063 2283 rebalancer_tool.cc:1136] tablet 0b8d119ee8474a40ac08df61df5b8423: '08d7a0680d244b55b93d17f9ab57b84c' -> '9dec645b473f40578e726f3e50e219bf' move completed: OK
I20240502 21:26:58.170943 2283 rebalancer_tool.cc:1136] tablet 46279270b2334633906af57c4b43e969: 'd1406e92aec84ac585304e57bd37a40d' -> '05f4fac41c4f4959b9b316cb112d0ada' move completed: OK
I20240502 21:26:58.310613 2283 rebalancer_tool.cc:1136] tablet fc93b5adc1d94e809e97e56b56822611: 'a44f41b0b6644fb6aa29476aed8c964f' -> '9dec645b473f40578e726f3e50e219bf' move completed: OK
I20240502 21:26:58.586846 2283 rebalancer_tool.cc:1136] tablet 5b1dcae186f542cea92c9191fa8ef951: '2d50f33557944bb480ea32199d9ee4f1' -> '1e04157b98dd4fad98858c290b73c3db' move completed: OK
I20240502 21:26:58.709232 2283 rebalancer_tool.cc:1136] tablet f5e82c0ab5c441efbcfbcd383b8db82f: '1ade0354c44c4c37ae480c7a94b11e1b' -> '08d7a0680d244b55b93d17f9ab57b84c' move completed: OK
I20240502 21:26:58.741693 2283 rebalancer_tool.cc:1074] tablet c5ba5610b754466fb1e12db451295809: '9dec645b473f40578e726f3e50e219bf' -> '08d7a0680d244b55b93d17f9ab57b84c' move scheduled
I20240502 21:26:58.965164 2283 rebalancer_tool.cc:1074] tablet 94ba99a5798f4d5c9442590066896f93: '2d50f33557944bb480ea32199d9ee4f1' -> '1ade0354c44c4c37ae480c7a94b11e1b' move scheduled
I20240502 21:26:59.167191 2283 rebalancer_tool.cc:1074] tablet b20c2bb60f6149458a62ee443b8cf3ab: 'd1406e92aec84ac585304e57bd37a40d' -> '05f4fac41c4f4959b9b316cb112d0ada' move scheduled
I20240502 21:26:59.370543 2283 rebalancer_tool.cc:1074] tablet 3e6657723fe84e749dea768de44638e7: '1e04157b98dd4fad98858c290b73c3db' -> '9dec645b473f40578e726f3e50e219bf' move scheduled
I20240502 21:26:59.956516 2283 rebalancer_tool.cc:1074] tablet a699f448e116428f85cfee6ccbc98612: '08d7a0680d244b55b93d17f9ab57b84c' -> '2d50f33557944bb480ea32199d9ee4f1' move scheduled
I20240502 21:27:00.084969 2283 rebalancer_tool.cc:1074] tablet dd0198178da640688230aeb5a1106cb4: 'a44f41b0b6644fb6aa29476aed8c964f' -> '1ade0354c44c4c37ae480c7a94b11e1b' move scheduled
I20240502 21:27:00.225311 2283 rebalancer_tool.cc:1074] tablet ae56d18c629a4e5eb50911c89de4227c: '05f4fac41c4f4959b9b316cb112d0ada' -> 'd1406e92aec84ac585304e57bd37a40d' move scheduled
I20240502 21:27:00.389366 2283 rebalancer_tool.cc:1074] tablet 9bc57b8a0388416290997df544e5cc0b: '1e04157b98dd4fad98858c290b73c3db' -> '08d7a0680d244b55b93d17f9ab57b84c' move scheduled
I20240502 21:27:00.422734 2283 rebalancer_tool.cc:1074] tablet 59f36d947f4341f39b6969cc1e065b6c: '1ade0354c44c4c37ae480c7a94b11e1b' -> '9dec645b473f40578e726f3e50e219bf' move scheduled
I20240502 21:27:00.480299 2283 rebalancer_tool.cc:1074] tablet 51551a1967b74094b373cc030e2bf850: '05f4fac41c4f4959b9b316cb112d0ada' -> 'a44f41b0b6644fb6aa29476aed8c964f' move scheduled
I20240502 21:27:00.511193 2283 rebalancer_tool.cc:1074] tablet f74ea10ab8a2459794f816707b87f494: 'd1406e92aec84ac585304e57bd37a40d' -> '08d7a0680d244b55b93d17f9ab57b84c' move scheduled
I20240502 21:27:01.672477 2283 rebalancer_tool.cc:1136] tablet b20c2bb60f6149458a62ee443b8cf3ab: 'd1406e92aec84ac585304e57bd37a40d' -> '05f4fac41c4f4959b9b316cb112d0ada' move completed: OK
I20240502 21:27:01.837982 2283 rebalancer_tool.cc:1136] tablet 94ba99a5798f4d5c9442590066896f93: '2d50f33557944bb480ea32199d9ee4f1' -> '1ade0354c44c4c37ae480c7a94b11e1b' move completed: OK
I20240502 21:27:02.025283 2283 rebalancer_tool.cc:1136] tablet c5ba5610b754466fb1e12db451295809: '9dec645b473f40578e726f3e50e219bf' -> '08d7a0680d244b55b93d17f9ab57b84c' move completed: OK
I20240502 21:27:02.048969 2283 rebalancer_tool.cc:1136] tablet 53b7eec870c44999b48f0bf744b2ffcd: 'a44f41b0b6644fb6aa29476aed8c964f' -> '1e04157b98dd4fad98858c290b73c3db' move completed: OK
I20240502 21:27:02.070338 2283 rebalancer_tool.cc:1136] tablet 0d923caa3f034a899050fe62dfc0c3d8: '9dec645b473f40578e726f3e50e219bf' -> '08d7a0680d244b55b93d17f9ab57b84c' move completed: OK
I20240502 21:27:02.207896 2283 rebalancer_tool.cc:1136] tablet 59f36d947f4341f39b6969cc1e065b6c: '1ade0354c44c4c37ae480c7a94b11e1b' -> '9dec645b473f40578e726f3e50e219bf' move completed: OK
I20240502 21:27:02.352022 2283 rebalancer_tool.cc:1136] tablet 2068521c26dc44f9ba60e274eb039b22: '1ade0354c44c4c37ae480c7a94b11e1b' -> 'a44f41b0b6644fb6aa29476aed8c964f' move completed: OK
I20240502 21:27:02.419759 2283 rebalancer_tool.cc:1136] tablet f13c3b9c4b7c471fa34156b781473ec4: 'd1406e92aec84ac585304e57bd37a40d' -> 'a44f41b0b6644fb6aa29476aed8c964f' move completed: OK
I20240502 21:27:02.502321 2283 rebalancer_tool.cc:1136] tablet e0f96b56fb68439a97cf2aae4aa89b57: '05f4fac41c4f4959b9b316cb112d0ada' -> '1ade0354c44c4c37ae480c7a94b11e1b' move completed: OK
I20240502 21:27:02.710772 2283 rebalancer_tool.cc:1136] tablet 534b4d96df9b4bc5a9480b6c45eb11a5: 'd1406e92aec84ac585304e57bd37a40d' -> '2d50f33557944bb480ea32199d9ee4f1' move completed: OK
I20240502 21:27:02.738869 2283 rebalancer_tool.cc:1074] tablet 5bc6c7211d5f4f73b2a050b9ec78c6b1: '1ade0354c44c4c37ae480c7a94b11e1b' -> '9dec645b473f40578e726f3e50e219bf' move scheduled
I20240502 21:27:02.813717 2283 rebalancer_tool.cc:1074] tablet f61e34e65429408e83a13c7bc743e78a: '2d50f33557944bb480ea32199d9ee4f1' -> '1ade0354c44c4c37ae480c7a94b11e1b' move scheduled
I20240502 21:27:02.848150 2283 rebalancer_tool.cc:1074] tablet 1f743c7345184feaa50871dc49a1f136: 'a44f41b0b6644fb6aa29476aed8c964f' -> '08d7a0680d244b55b93d17f9ab57b84c' move scheduled
I20240502 21:27:02.898587 2283 rebalancer_tool.cc:1074] tablet 4bc968ae4dcc4636a49257b6d5824bee: '9dec645b473f40578e726f3e50e219bf' -> '05f4fac41c4f4959b9b316cb112d0ada' move scheduled
I20240502 21:27:02.942775 2283 rebalancer_tool.cc:1074] tablet 883c729aa26f47178f40a17ad3b7e590: '1ade0354c44c4c37ae480c7a94b11e1b' -> '2d50f33557944bb480ea32199d9ee4f1' move scheduled
I20240502 21:27:02.972018 2283 rebalancer_tool.cc:1074] tablet 381ac1d714204cef86c8e1a73e24b863: '08d7a0680d244b55b93d17f9ab57b84c' -> '1e04157b98dd4fad98858c290b73c3db' move scheduled
I20240502 21:27:03.006698 2283 rebalancer_tool.cc:1074] tablet b8db5de7df864ab983504e96fb738106: '9dec645b473f40578e726f3e50e219bf' -> '05f4fac41c4f4959b9b316cb112d0ada' move scheduled
I20240502 21:27:04.027120 2283 rebalancer_tool.cc:1136] tablet f74ea10ab8a2459794f816707b87f494: 'd1406e92aec84ac585304e57bd37a40d' -> '08d7a0680d244b55b93d17f9ab57b84c' move completed: OK
I20240502 21:27:04.052187 2283 rebalancer_tool.cc:1136] tablet 51551a1967b74094b373cc030e2bf850: '05f4fac41c4f4959b9b316cb112d0ada' -> 'a44f41b0b6644fb6aa29476aed8c964f' move completed: OK
I20240502 21:27:04.105707 2283 rebalancer_tool.cc:1136] tablet ae56d18c629a4e5eb50911c89de4227c: '05f4fac41c4f4959b9b316cb112d0ada' -> 'd1406e92aec84ac585304e57bd37a40d' move completed: OK
I20240502 21:27:04.143136 2283 rebalancer_tool.cc:1136] tablet a699f448e116428f85cfee6ccbc98612: '08d7a0680d244b55b93d17f9ab57b84c' -> '2d50f33557944bb480ea32199d9ee4f1' move completed: OK
I20240502 21:27:04.193321 2283 rebalancer_tool.cc:1136] tablet 2c737c49a32b4f3ea5ef735fa254d48e: '1e04157b98dd4fad98858c290b73c3db' -> '05f4fac41c4f4959b9b316cb112d0ada' move completed: OK
I20240502 21:27:04.344113 2283 rebalancer_tool.cc:1136] tablet 3e6657723fe84e749dea768de44638e7: '1e04157b98dd4fad98858c290b73c3db' -> '9dec645b473f40578e726f3e50e219bf' move completed: OK
I20240502 21:27:04.860726 2283 rebalancer_tool.cc:1074] tablet 62e70b8e69c94d76b23c655a3f5de5bd: '05f4fac41c4f4959b9b316cb112d0ada' -> '1ade0354c44c4c37ae480c7a94b11e1b' move scheduled
I20240502 21:27:05.313890 2283 rebalancer_tool.cc:1074] tablet 988a0a9425434cd0b724710e9ebf23a6: '08d7a0680d244b55b93d17f9ab57b84c' -> '1e04157b98dd4fad98858c290b73c3db' move scheduled
I20240502 21:27:05.589021 2283 rebalancer_tool.cc:1074] tablet 043f33cb58084c3ebbce4893d1b63fbc: '08d7a0680d244b55b93d17f9ab57b84c' -> '9dec645b473f40578e726f3e50e219bf' move scheduled
I20240502 21:27:05.810423 2283 rebalancer_tool.cc:1136] tablet b8db5de7df864ab983504e96fb738106: '9dec645b473f40578e726f3e50e219bf' -> '05f4fac41c4f4959b9b316cb112d0ada' move completed: OK
I20240502 21:27:05.869657 2283 rebalancer_tool.cc:1136] tablet 4bc968ae4dcc4636a49257b6d5824bee: '9dec645b473f40578e726f3e50e219bf' -> '05f4fac41c4f4959b9b316cb112d0ada' move completed: OK
I20240502 21:27:05.923472 2283 rebalancer_tool.cc:1136] tablet 1f743c7345184feaa50871dc49a1f136: 'a44f41b0b6644fb6aa29476aed8c964f' -> '08d7a0680d244b55b93d17f9ab57b84c' move completed: OK
I20240502 21:27:05.934562 2283 rebalancer_tool.cc:1136] tablet f61e34e65429408e83a13c7bc743e78a: '2d50f33557944bb480ea32199d9ee4f1' -> '1ade0354c44c4c37ae480c7a94b11e1b' move completed: OK
I20240502 21:27:05.943214 2283 rebalancer_tool.cc:1136] tablet 381ac1d714204cef86c8e1a73e24b863: '08d7a0680d244b55b93d17f9ab57b84c' -> '1e04157b98dd4fad98858c290b73c3db' move completed: OK
I20240502 21:27:05.951689 2283 rebalancer_tool.cc:1136] tablet 5bc6c7211d5f4f73b2a050b9ec78c6b1: '1ade0354c44c4c37ae480c7a94b11e1b' -> '9dec645b473f40578e726f3e50e219bf' move completed: OK
I20240502 21:27:05.963606 2283 rebalancer_tool.cc:1136] tablet 9bc57b8a0388416290997df544e5cc0b: '1e04157b98dd4fad98858c290b73c3db' -> '08d7a0680d244b55b93d17f9ab57b84c' move completed: OK
I20240502 21:27:05.978874 2283 rebalancer_tool.cc:1136] tablet dd0198178da640688230aeb5a1106cb4: 'a44f41b0b6644fb6aa29476aed8c964f' -> '1ade0354c44c4c37ae480c7a94b11e1b' move completed: OK
I20240502 21:27:05.986312 2283 rebalancer_tool.cc:1136] tablet 883c729aa26f47178f40a17ad3b7e590: '1ade0354c44c4c37ae480c7a94b11e1b' -> '2d50f33557944bb480ea32199d9ee4f1' move completed: OK
I20240502 21:27:06.057719 2283 rebalancer_tool.cc:1074] tablet 04c07a003bfe472e9142df6082759ae1: '1ade0354c44c4c37ae480c7a94b11e1b' -> '05f4fac41c4f4959b9b316cb112d0ada' move scheduled
I20240502 21:27:06.237674 2283 rebalancer_tool.cc:1074] tablet 88189733d9a14560971dbeb71543073f: '08d7a0680d244b55b93d17f9ab57b84c' -> '9dec645b473f40578e726f3e50e219bf' move scheduled
I20240502 21:27:06.333709 2283 rebalancer_tool.cc:1074] tablet 5a54d2aad63c4f3182992f00b8564acf: '08d7a0680d244b55b93d17f9ab57b84c' -> '1e04157b98dd4fad98858c290b73c3db' move scheduled
W20240502 21:27:06.681608 5925 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 21:27:06.401619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.226.134:43753 (local address 127.0.0.1:35502)
0502 21:27:06.402396 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0502 21:27:06.402416 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0502 21:27:06.402538 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 21:27:06.676306 (+273768us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":487,"threads_started":1}
I20240502 21:27:06.688418 2283 rebalancer_tool.cc:1084] tablet f02f83ec0a494da5b66241d9b379a03f: '1ade0354c44c4c37ae480c7a94b11e1b' -> '05f4fac41c4f4959b9b316cb112d0ada' move ignored: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:27:06.688612 2283 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): unacceptable health status UNAVAILABLE
I20240502 21:27:09.317042 31797 raft_consensus.cc:2454] T 056db9c270e94ba58d356f7eaff5c124 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:27:09.318085 2092 raft_consensus.cc:2454] T 056db9c270e94ba58d356f7eaff5c124 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:27:09.318655 31471 leader_election.cc:302] T 056db9c270e94ba58d356f7eaff5c124 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:27:09.318914 6024 raft_consensus.cc:2781] T 056db9c270e94ba58d356f7eaff5c124 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:09.319806 6024 raft_consensus.cc:686] T 056db9c270e94ba58d356f7eaff5c124 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:27:09.320142 5705 leader_election.cc:288] T 056db9c270e94ba58d356f7eaff5c124 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Requested vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:27:09.320073 6024 consensus_queue.cc:237] T 056db9c270e94ba58d356f7eaff5c124 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:09.321849 6072 raft_consensus.cc:482] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:09.321974 6072 raft_consensus.cc:504] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:09.322593 6072 leader_election.cc:288] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:27:09.323000 1834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1ff0b7b574c05b15ff6e8aa9a3c54" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:27:09.323210 1834 raft_consensus.cc:2454] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:27:09.323552 31602 leader_election.cc:302] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 1e04157b98dd4fad98858c290b73c3db; no voters:
I20240502 21:27:09.323516 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1ff0b7b574c05b15ff6e8aa9a3c54" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:27:09.323714 31797 raft_consensus.cc:2454] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:27:09.323777 6072 raft_consensus.cc:2781] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:09.323854 6072 raft_consensus.cc:482] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:09.323912 6072 raft_consensus.cc:3037] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:09.323706 31394 catalog_manager.cc:5484] T 056db9c270e94ba58d356f7eaff5c124 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 0 to 1, leader changed from <none> to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:09.326016 6072 raft_consensus.cc:504] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:09.326493 6072 leader_election.cc:288] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Requested vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:27:09.327106 1831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1ff0b7b574c05b15ff6e8aa9a3c54" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:27:09.327296 1831 raft_consensus.cc:3037] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:09.327674 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b1ff0b7b574c05b15ff6e8aa9a3c54" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:27:09.327997 31797 raft_consensus.cc:3037] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:09.329448 1831 raft_consensus.cc:2454] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:27:09.330945 31602 leader_election.cc:302] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 1e04157b98dd4fad98858c290b73c3db; no voters:
I20240502 21:27:09.331177 6072 raft_consensus.cc:2781] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:09.331555 6072 raft_consensus.cc:686] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Becoming Leader. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Running, Role: LEADER
I20240502 21:27:09.331811 6072 consensus_queue.cc:237] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:09.332500 31797 raft_consensus.cc:2454] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:27:09.334983 31394 catalog_manager.cc:5484] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130). New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:09.373495 31394 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:09.381219 6190 tablet.cc:1722] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:09.382099 6191 tablet.cc:1722] T 97a450582be947a48b4aa789389e9e5a P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:09.389083 31394 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table_new_name [id=2dcceb9804094dee8643679132a7ba30] alter complete (version 3)
W20240502 21:27:09.425511 1767 consensus_peers.cc:485] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:27:09.492220 31388 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:09.492358 31388 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:09.494983 31388 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 97a450582be947a48b4aa789389e9e5a
I20240502 21:27:09.495852 1814 tablet_service.cc:1514] Processing DeleteTablet for tablet 97a450582be947a48b4aa789389e9e5a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:09 UTC) from {username='slave'} at 127.0.0.1:47510
I20240502 21:27:09.495905 2077 tablet_service.cc:1514] Processing DeleteTablet for tablet 97a450582be947a48b4aa789389e9e5a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:09 UTC) from {username='slave'} at 127.0.0.1:36740
I20240502 21:27:09.496464 6209 tablet_replica.cc:330] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b: stopping tablet replica
I20240502 21:27:09.496721 6209 raft_consensus.cc:2229] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:09.496958 6210 tablet_replica.cc:330] T 97a450582be947a48b4aa789389e9e5a P 1e04157b98dd4fad98858c290b73c3db: stopping tablet replica
I20240502 21:27:09.497193 6210 raft_consensus.cc:2229] T 97a450582be947a48b4aa789389e9e5a P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:09.497174 6209 raft_consensus.cc:2258] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:09.497455 6210 raft_consensus.cc:2258] T 97a450582be947a48b4aa789389e9e5a P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:09.497908 6209 ts_tablet_manager.cc:1875] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:09.498150 6210 ts_tablet_manager.cc:1875] T 97a450582be947a48b4aa789389e9e5a P 1e04157b98dd4fad98858c290b73c3db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:09.504017 6210 ts_tablet_manager.cc:1888] T 97a450582be947a48b4aa789389e9e5a P 1e04157b98dd4fad98858c290b73c3db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:09.504016 6209 ts_tablet_manager.cc:1888] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:09.504142 6209 log.cc:1196] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal/wals/97a450582be947a48b4aa789389e9e5a
I20240502 21:27:09.504153 6210 log.cc:1196] T 97a450582be947a48b4aa789389e9e5a P 1e04157b98dd4fad98858c290b73c3db: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal/wals/97a450582be947a48b4aa789389e9e5a
I20240502 21:27:09.504644 6209 ts_tablet_manager.cc:1909] T 97a450582be947a48b4aa789389e9e5a P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting consensus metadata
I20240502 21:27:09.504643 6210 ts_tablet_manager.cc:1909] T 97a450582be947a48b4aa789389e9e5a P 1e04157b98dd4fad98858c290b73c3db: Deleting consensus metadata
I20240502 21:27:09.505548 31381 catalog_manager.cc:4830] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): tablet 97a450582be947a48b4aa789389e9e5a (table rebalancer_dynamic_table_new_name [id=2dcceb9804094dee8643679132a7ba30]) successfully deleted
I20240502 21:27:09.505774 31381 catalog_manager.cc:4830] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): tablet 97a450582be947a48b4aa789389e9e5a (table rebalancer_dynamic_table_new_name [id=2dcceb9804094dee8643679132a7ba30]) successfully deleted
I20240502 21:27:09.597633 31388 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:09.603775 1814 tablet_service.cc:1467] Processing CreateTablet for tablet ee39947f94214af8ac16d650c467363e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9fc9a8dd5172486398d9d08dda61f943]), partition=
I20240502 21:27:09.603893 31647 tablet_service.cc:1467] Processing CreateTablet for tablet ee39947f94214af8ac16d650c467363e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9fc9a8dd5172486398d9d08dda61f943]), partition=
I20240502 21:27:09.604107 1814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee39947f94214af8ac16d650c467363e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:09.604224 31647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee39947f94214af8ac16d650c467363e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:09.607197 31906 tablet_service.cc:1467] Processing CreateTablet for tablet ee39947f94214af8ac16d650c467363e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9fc9a8dd5172486398d9d08dda61f943]), partition=
I20240502 21:27:09.607524 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee39947f94214af8ac16d650c467363e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:09.609869 6194 tablet_bootstrap.cc:492] T ee39947f94214af8ac16d650c467363e P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:27:09.610944 6194 tablet_bootstrap.cc:654] T ee39947f94214af8ac16d650c467363e P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:09.616036 6194 tablet_bootstrap.cc:492] T ee39947f94214af8ac16d650c467363e P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:27:09.616194 6194 ts_tablet_manager.cc:1367] T ee39947f94214af8ac16d650c467363e P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20240502 21:27:09.616966 6172 tablet_bootstrap.cc:492] T ee39947f94214af8ac16d650c467363e P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:27:09.617475 6212 tablet_bootstrap.cc:492] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:27:09.617450 6194 raft_consensus.cc:348] T ee39947f94214af8ac16d650c467363e P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:09.617620 6194 raft_consensus.cc:374] T ee39947f94214af8ac16d650c467363e P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:09.617695 6194 raft_consensus.cc:729] T ee39947f94214af8ac16d650c467363e P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:27:09.617926 6194 consensus_queue.cc:260] T ee39947f94214af8ac16d650c467363e P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:09.618252 6172 tablet_bootstrap.cc:654] T ee39947f94214af8ac16d650c467363e P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:09.618403 6194 ts_tablet_manager.cc:1398] T ee39947f94214af8ac16d650c467363e P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:09.618638 6212 tablet_bootstrap.cc:654] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:09.620208 6212 tablet_bootstrap.cc:492] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:27:09.620455 6212 ts_tablet_manager.cc:1367] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 21:27:09.621515 6072 raft_consensus.cc:482] T 92565312f28a4e3fb0c20a0d0b03f24f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:09.621665 6072 raft_consensus.cc:504] T 92565312f28a4e3fb0c20a0d0b03f24f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:09.621860 6212 raft_consensus.cc:348] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:09.622077 6212 raft_consensus.cc:374] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:09.622174 6212 raft_consensus.cc:729] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:27:09.622222 6072 leader_election.cc:288] T 92565312f28a4e3fb0c20a0d0b03f24f P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:27:09.622448 6212 consensus_queue.cc:260] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:09.622853 2093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92565312f28a4e3fb0c20a0d0b03f24f" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:27:09.622903 1831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92565312f28a4e3fb0c20a0d0b03f24f" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:27:09.623096 1831 raft_consensus.cc:2454] T 92565312f28a4e3fb0c20a0d0b03f24f P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:27:09.623097 2093 raft_consensus.cc:2454] T 92565312f28a4e3fb0c20a0d0b03f24f P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:27:09.623466 31602 leader_election.cc:302] T 92565312f28a4e3fb0c20a0d0b03f24f P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 1e04157b98dd4fad98858c290b73c3db; no voters:
I20240502 21:27:09.623574 6212 ts_tablet_manager.cc:1398] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 21:27:09.623719 6072 raft_consensus.cc:2781] T 92565312f28a4e3fb0c20a0d0b03f24f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:09.623754 6172 tablet_bootstrap.cc:492] T ee39947f94214af8ac16d650c467363e P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:27:09.623833 6072 raft_consensus.cc:482] T 92565312f28a4e3fb0c20a0d0b03f24f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:09.623894 6172 ts_tablet_manager.cc:1367] T ee39947f94214af8ac16d650c467363e P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240502 21:27:09.623946 6072 raft_consensus.cc:3037] T 92565312f28a4e3fb0c20a0d0b03f24f P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:09.624567 6172 raft_consensus.cc:348] T ee39947f94214af8ac16d650c467363e P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:09.624781 6172 raft_consensus.cc:374] T ee39947f94214af8ac16d650c467363e P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:09.625264 6172 raft_consensus.cc:729] T ee39947f94214af8ac16d650c467363e P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:27:09.626180 6172 consensus_queue.cc:260] T ee39947f94214af8ac16d650c467363e P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:09.626405 6072 raft_consensus.cc:504] T 92565312f28a4e3fb0c20a0d0b03f24f P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:09.626996 6172 ts_tablet_manager.cc:1398] T ee39947f94214af8ac16d650c467363e P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:09.627628 1831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92565312f28a4e3fb0c20a0d0b03f24f" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:27:09.627847 1831 raft_consensus.cc:3037] T 92565312f28a4e3fb0c20a0d0b03f24f P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:09.628019 6072 leader_election.cc:288] T 92565312f28a4e3fb0c20a0d0b03f24f P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Requested vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:27:09.628342 2093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92565312f28a4e3fb0c20a0d0b03f24f" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:27:09.628528 2093 raft_consensus.cc:3037] T 92565312f28a4e3fb0c20a0d0b03f24f P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:09.630162 1831 raft_consensus.cc:2454] T 92565312f28a4e3fb0c20a0d0b03f24f P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:27:09.630736 31602 leader_election.cc:302] T 92565312f28a4e3fb0c20a0d0b03f24f P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 1e04157b98dd4fad98858c290b73c3db; no voters:
I20240502 21:27:09.630987 6072 raft_consensus.cc:2781] T 92565312f28a4e3fb0c20a0d0b03f24f P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:09.631168 2093 raft_consensus.cc:2454] T 92565312f28a4e3fb0c20a0d0b03f24f P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:27:09.631429 6072 raft_consensus.cc:686] T 92565312f28a4e3fb0c20a0d0b03f24f P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Becoming Leader. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Running, Role: LEADER
I20240502 21:27:09.631664 6072 consensus_queue.cc:237] T 92565312f28a4e3fb0c20a0d0b03f24f P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:09.637171 31393 catalog_manager.cc:5484] T 92565312f28a4e3fb0c20a0d0b03f24f P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130). New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:09.723114 6072 consensus_queue.cc:1035] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:09.723560 6205 consensus_queue.cc:1035] T 056db9c270e94ba58d356f7eaff5c124 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:09.742055 31600 consensus_peers.cc:462] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:09.745841 6024 consensus_queue.cc:1035] T 056db9c270e94ba58d356f7eaff5c124 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:09.747512 6072 consensus_queue.cc:1035] T 54b1ff0b7b574c05b15ff6e8aa9a3c54 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:09.789078 5896 raft_consensus.cc:482] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:09.789223 5896 raft_consensus.cc:504] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:09.789757 5896 leader_election.cc:288] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:27:09.790180 1831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee39947f94214af8ac16d650c467363e" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:27:09.790371 1831 raft_consensus.cc:2454] T ee39947f94214af8ac16d650c467363e P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:27:09.790683 31862 leader_election.cc:302] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:27:09.790810 31664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee39947f94214af8ac16d650c467363e" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:27:09.790906 5896 raft_consensus.cc:2781] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:09.790969 5896 raft_consensus.cc:482] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:09.791000 31664 raft_consensus.cc:2454] T ee39947f94214af8ac16d650c467363e P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 0.
I20240502 21:27:09.791035 5896 raft_consensus.cc:3037] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:09.793426 5896 raft_consensus.cc:504] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:09.793959 5896 leader_election.cc:288] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Requested vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:27:09.794313 1831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee39947f94214af8ac16d650c467363e" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:27:09.794474 1831 raft_consensus.cc:3037] T ee39947f94214af8ac16d650c467363e P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:09.796797 31664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee39947f94214af8ac16d650c467363e" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:27:09.796967 31664 raft_consensus.cc:3037] T ee39947f94214af8ac16d650c467363e P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:09.797164 1831 raft_consensus.cc:2454] T ee39947f94214af8ac16d650c467363e P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:27:09.797544 31862 leader_election.cc:302] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, 2d50f33557944bb480ea32199d9ee4f1; no voters:
I20240502 21:27:09.797820 5896 raft_consensus.cc:2781] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:09.797988 5896 raft_consensus.cc:686] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Running, Role: LEADER
I20240502 21:27:09.798270 5896 consensus_queue.cc:237] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:09.799463 31664 raft_consensus.cc:2454] T ee39947f94214af8ac16d650c467363e P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:27:09.802824 31388 catalog_manager.cc:5484] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132). New cstate: current_term: 1 leader_uuid: "2d50f33557944bb480ea32199d9ee4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: HEALTHY } } }
W20240502 21:27:09.804425 31379 catalog_manager.cc:4573] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 97a450582be947a48b4aa789389e9e5a: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:09.921697 31388 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:09.926378 31664 raft_consensus.cc:1261] T ee39947f94214af8ac16d650c467363e P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:09.926407 1831 raft_consensus.cc:1261] T ee39947f94214af8ac16d650c467363e P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer 2d50f33557944bb480ea32199d9ee4f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:09.926962 5896 consensus_queue.cc:1035] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:09.927232 6208 consensus_queue.cc:1035] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:09.931707 6228 tablet.cc:1722] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:09.932278 6228 tablet.cc:1620] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1: MemRowSet was empty: no flush needed.
I20240502 21:27:09.933301 31394 catalog_manager.cc:5576] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645) does not have the latest schema for tablet ee39947f94214af8ac16d650c467363e (table rebalancer_dynamic_table [id=9fc9a8dd5172486398d9d08dda61f943]). Expected version 1 got 0
I20240502 21:27:09.935811 31393 catalog_manager.cc:5576] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933) does not have the latest schema for tablet ee39947f94214af8ac16d650c467363e (table rebalancer_dynamic_table [id=9fc9a8dd5172486398d9d08dda61f943]). Expected version 1 got 0
I20240502 21:27:09.942097 6229 tablet.cc:1722] T ee39947f94214af8ac16d650c467363e P 05f4fac41c4f4959b9b316cb112d0ada: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:09.942586 6229 tablet.cc:1620] T ee39947f94214af8ac16d650c467363e P 05f4fac41c4f4959b9b316cb112d0ada: MemRowSet was empty: no flush needed.
I20240502 21:27:09.947968 31393 catalog_manager.cc:5576] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645) does not have the latest schema for tablet ee39947f94214af8ac16d650c467363e (table rebalancer_dynamic_table [id=9fc9a8dd5172486398d9d08dda61f943]). Expected version 1 got 0
I20240502 21:27:09.949190 31394 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=9fc9a8dd5172486398d9d08dda61f943] alter complete (version 1)
I20240502 21:27:09.950212 31388 catalog_manager.cc:5576] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863) does not have the latest schema for tablet ee39947f94214af8ac16d650c467363e (table rebalancer_dynamic_table [id=9fc9a8dd5172486398d9d08dda61f943]). Expected version 1 got 0
I20240502 21:27:09.952495 6230 tablet.cc:1722] T ee39947f94214af8ac16d650c467363e P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:09.953008 6230 tablet.cc:1620] T ee39947f94214af8ac16d650c467363e P 1e04157b98dd4fad98858c290b73c3db: MemRowSet was empty: no flush needed.
I20240502 21:27:09.955382 31394 catalog_manager.cc:5576] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863) does not have the latest schema for tablet ee39947f94214af8ac16d650c467363e (table rebalancer_dynamic_table [id=9fc9a8dd5172486398d9d08dda61f943]). Expected version 1 got 0
W20240502 21:27:10.006099 31600 consensus_peers.cc:485] T eed5a5c41d9a46ce8a712b5b399e0472 P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:27:10.019663 6072 consensus_queue.cc:1035] T 92565312f28a4e3fb0c20a0d0b03f24f P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:10.024000 6021 consensus_queue.cc:1035] T 92565312f28a4e3fb0c20a0d0b03f24f P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:10.054090 31393 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:10.061398 6228 tablet.cc:1722] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:10.061866 6228 tablet.cc:1620] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1: MemRowSet was empty: no flush needed.
I20240502 21:27:10.062511 6230 tablet.cc:1722] T ee39947f94214af8ac16d650c467363e P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:10.063028 6230 tablet.cc:1620] T ee39947f94214af8ac16d650c467363e P 1e04157b98dd4fad98858c290b73c3db: MemRowSet was empty: no flush needed.
I20240502 21:27:10.063482 6229 tablet.cc:1722] T ee39947f94214af8ac16d650c467363e P 05f4fac41c4f4959b9b316cb112d0ada: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:10.063931 6229 tablet.cc:1620] T ee39947f94214af8ac16d650c467363e P 05f4fac41c4f4959b9b316cb112d0ada: MemRowSet was empty: no flush needed.
I20240502 21:27:10.072010 31393 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=9fc9a8dd5172486398d9d08dda61f943] alter complete (version 2)
I20240502 21:27:10.145356 31393 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_extra_table_52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:10.154217 31906 tablet_service.cc:1467] Processing CreateTablet for tablet 3d024593597a4230b5c25b806843b779 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=d785ac70ed3040dab50f8bfe463fed07]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:10.154368 31514 tablet_service.cc:1467] Processing CreateTablet for tablet d702ad56b7e149e69fa6d1f33f78c175 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=d785ac70ed3040dab50f8bfe463fed07]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:10.154656 31514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d702ad56b7e149e69fa6d1f33f78c175. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:10.154729 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d024593597a4230b5c25b806843b779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:10.156092 2210 tablet_service.cc:1467] Processing CreateTablet for tablet 3d024593597a4230b5c25b806843b779 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=d785ac70ed3040dab50f8bfe463fed07]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:10.156108 31774 tablet_service.cc:1467] Processing CreateTablet for tablet fb101c0103ec4374b9ca17dea02bf0c4 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=d785ac70ed3040dab50f8bfe463fed07]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:10.156512 31774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb101c0103ec4374b9ca17dea02bf0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:10.156539 2210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d024593597a4230b5c25b806843b779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:10.156832 31775 tablet_service.cc:1467] Processing CreateTablet for tablet 3d024593597a4230b5c25b806843b779 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=d785ac70ed3040dab50f8bfe463fed07]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:10.157196 31775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d024593597a4230b5c25b806843b779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:10.159256 2208 tablet_service.cc:1467] Processing CreateTablet for tablet d702ad56b7e149e69fa6d1f33f78c175 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=d785ac70ed3040dab50f8bfe463fed07]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:10.159615 2208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d702ad56b7e149e69fa6d1f33f78c175. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:10.160761 2209 tablet_service.cc:1467] Processing CreateTablet for tablet fb101c0103ec4374b9ca17dea02bf0c4 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=d785ac70ed3040dab50f8bfe463fed07]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:10.160784 1814 tablet_service.cc:1467] Processing CreateTablet for tablet fb101c0103ec4374b9ca17dea02bf0c4 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=d785ac70ed3040dab50f8bfe463fed07]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:10.161157 1814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb101c0103ec4374b9ca17dea02bf0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:10.161161 2209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb101c0103ec4374b9ca17dea02bf0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:10.166854 2077 tablet_service.cc:1467] Processing CreateTablet for tablet d702ad56b7e149e69fa6d1f33f78c175 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=d785ac70ed3040dab50f8bfe463fed07]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:10.167260 2077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d702ad56b7e149e69fa6d1f33f78c175. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:10.172595 6238 tablet_bootstrap.cc:492] T 3d024593597a4230b5c25b806843b779 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:27:10.174778 6237 tablet_bootstrap.cc:492] T d702ad56b7e149e69fa6d1f33f78c175 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:27:10.176627 6237 tablet_bootstrap.cc:654] T d702ad56b7e149e69fa6d1f33f78c175 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:10.176513 31394 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:10.177022 6238 tablet_bootstrap.cc:654] T 3d024593597a4230b5c25b806843b779 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:10.177070 6236 tablet_bootstrap.cc:492] T 3d024593597a4230b5c25b806843b779 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:27:10.179961 6236 tablet_bootstrap.cc:654] T 3d024593597a4230b5c25b806843b779 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:10.184012 6228 tablet.cc:1722] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:10.185727 6230 tablet.cc:1722] T ee39947f94214af8ac16d650c467363e P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:10.187108 6229 tablet.cc:1722] T ee39947f94214af8ac16d650c467363e P 05f4fac41c4f4959b9b316cb112d0ada: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:10.198609 31394 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table_new_name [id=9fc9a8dd5172486398d9d08dda61f943] alter complete (version 3)
I20240502 21:27:10.215432 6242 tablet_bootstrap.cc:492] T fb101c0103ec4374b9ca17dea02bf0c4 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:27:10.215483 6243 tablet_bootstrap.cc:492] T d702ad56b7e149e69fa6d1f33f78c175 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:27:10.216738 6235 tablet_bootstrap.cc:492] T 3d024593597a4230b5c25b806843b779 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:27:10.216914 6238 tablet_bootstrap.cc:492] T 3d024593597a4230b5c25b806843b779 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:27:10.217134 6238 ts_tablet_manager.cc:1367] T 3d024593597a4230b5c25b806843b779 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.045s user 0.010s sys 0.006s
I20240502 21:27:10.217828 6238 raft_consensus.cc:348] T 3d024593597a4230b5c25b806843b779 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:10.218015 6238 raft_consensus.cc:374] T 3d024593597a4230b5c25b806843b779 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:10.218086 6238 raft_consensus.cc:729] T 3d024593597a4230b5c25b806843b779 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:27:10.218375 6242 tablet_bootstrap.cc:654] T fb101c0103ec4374b9ca17dea02bf0c4 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:10.218286 6238 consensus_queue.cc:260] T 3d024593597a4230b5c25b806843b779 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:10.218827 6238 ts_tablet_manager.cc:1398] T 3d024593597a4230b5c25b806843b779 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:10.219373 6004 raft_consensus.cc:482] T 3d024593597a4230b5c25b806843b779 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:10.219477 6004 raft_consensus.cc:504] T 3d024593597a4230b5c25b806843b779 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:10.220026 6004 leader_election.cc:288] T 3d024593597a4230b5c25b806843b779 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:27:10.220798 6238 tablet_bootstrap.cc:492] T fb101c0103ec4374b9ca17dea02bf0c4 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:27:10.220763 31923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d024593597a4230b5c25b806843b779" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:27:10.220939 2224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d024593597a4230b5c25b806843b779" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
W20240502 21:27:10.221329 31733 leader_election.cc:341] T 3d024593597a4230b5c25b806843b779 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:27:10.221580 31732 leader_election.cc:341] T 3d024593597a4230b5c25b806843b779 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:10.221673 31732 leader_election.cc:302] T 3d024593597a4230b5c25b806843b779 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c; no voters: 2d50f33557944bb480ea32199d9ee4f1, d1406e92aec84ac585304e57bd37a40d
I20240502 21:27:10.222048 6235 tablet_bootstrap.cc:654] T 3d024593597a4230b5c25b806843b779 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:10.222173 6238 tablet_bootstrap.cc:654] T fb101c0103ec4374b9ca17dea02bf0c4 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:10.222739 6004 raft_consensus.cc:2726] T 3d024593597a4230b5c25b806843b779 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:27:10.223073 6243 tablet_bootstrap.cc:654] T d702ad56b7e149e69fa6d1f33f78c175 P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:10.224710 6237 tablet_bootstrap.cc:492] T d702ad56b7e149e69fa6d1f33f78c175 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:27:10.224944 6237 ts_tablet_manager.cc:1367] T d702ad56b7e149e69fa6d1f33f78c175 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.008s
I20240502 21:27:10.225914 6237 raft_consensus.cc:348] T d702ad56b7e149e69fa6d1f33f78c175 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:10.226117 6237 raft_consensus.cc:374] T d702ad56b7e149e69fa6d1f33f78c175 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:10.226189 6237 raft_consensus.cc:729] T d702ad56b7e149e69fa6d1f33f78c175 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:27:10.226481 6242 tablet_bootstrap.cc:492] T fb101c0103ec4374b9ca17dea02bf0c4 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:27:10.226413 6237 consensus_queue.cc:260] T d702ad56b7e149e69fa6d1f33f78c175 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:10.226712 6242 ts_tablet_manager.cc:1367] T fb101c0103ec4374b9ca17dea02bf0c4 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.003s
I20240502 21:27:10.227483 6242 raft_consensus.cc:348] T fb101c0103ec4374b9ca17dea02bf0c4 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:10.227818 6242 raft_consensus.cc:374] T fb101c0103ec4374b9ca17dea02bf0c4 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:10.227932 6242 raft_consensus.cc:729] T fb101c0103ec4374b9ca17dea02bf0c4 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:27:10.228173 6242 consensus_queue.cc:260] T fb101c0103ec4374b9ca17dea02bf0c4 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:10.228886 6242 ts_tablet_manager.cc:1398] T fb101c0103ec4374b9ca17dea02bf0c4 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:10.229521 6204 raft_consensus.cc:482] T d702ad56b7e149e69fa6d1f33f78c175 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:10.229667 6204 raft_consensus.cc:504] T d702ad56b7e149e69fa6d1f33f78c175 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:10.230356 6204 leader_election.cc:288] T d702ad56b7e149e69fa6d1f33f78c175 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:27:10.230991 2224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d702ad56b7e149e69fa6d1f33f78c175" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:27:10.231055 2092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d702ad56b7e149e69fa6d1f33f78c175" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
W20240502 21:27:10.231511 31471 leader_election.cc:341] T d702ad56b7e149e69fa6d1f33f78c175 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:27:10.231701 31471 leader_election.cc:341] T d702ad56b7e149e69fa6d1f33f78c175 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:10.231781 31471 leader_election.cc:302] T d702ad56b7e149e69fa6d1f33f78c175 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a44f41b0b6644fb6aa29476aed8c964f; no voters: 1ade0354c44c4c37ae480c7a94b11e1b, d1406e92aec84ac585304e57bd37a40d
I20240502 21:27:10.232682 6204 raft_consensus.cc:2726] T d702ad56b7e149e69fa6d1f33f78c175 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:27:10.235574 6237 ts_tablet_manager.cc:1398] T d702ad56b7e149e69fa6d1f33f78c175 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 21:27:10.240813 6236 tablet_bootstrap.cc:492] T 3d024593597a4230b5c25b806843b779 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:27:10.241042 6236 ts_tablet_manager.cc:1367] T 3d024593597a4230b5c25b806843b779 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.064s user 0.010s sys 0.001s
I20240502 21:27:10.244307 6236 raft_consensus.cc:348] T 3d024593597a4230b5c25b806843b779 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:10.246888 6236 raft_consensus.cc:374] T 3d024593597a4230b5c25b806843b779 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:10.247033 6236 raft_consensus.cc:729] T 3d024593597a4230b5c25b806843b779 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:27:10.247746 6235 tablet_bootstrap.cc:492] T 3d024593597a4230b5c25b806843b779 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:27:10.247678 6236 consensus_queue.cc:260] T 3d024593597a4230b5c25b806843b779 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:10.248204 6236 ts_tablet_manager.cc:1398] T 3d024593597a4230b5c25b806843b779 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 21:27:10.250257 6235 ts_tablet_manager.cc:1367] T 3d024593597a4230b5c25b806843b779 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.011s
I20240502 21:27:10.250563 6243 tablet_bootstrap.cc:492] T d702ad56b7e149e69fa6d1f33f78c175 P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:27:10.250731 6243 ts_tablet_manager.cc:1367] T d702ad56b7e149e69fa6d1f33f78c175 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20240502 21:27:10.251019 6235 raft_consensus.cc:348] T 3d024593597a4230b5c25b806843b779 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:10.251194 6235 raft_consensus.cc:374] T 3d024593597a4230b5c25b806843b779 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:10.251340 6235 raft_consensus.cc:729] T 3d024593597a4230b5c25b806843b779 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:27:10.251559 6235 consensus_queue.cc:260] T 3d024593597a4230b5c25b806843b779 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:10.251626 6243 raft_consensus.cc:348] T d702ad56b7e149e69fa6d1f33f78c175 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:10.251889 6243 raft_consensus.cc:374] T d702ad56b7e149e69fa6d1f33f78c175 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:10.252005 6243 raft_consensus.cc:729] T d702ad56b7e149e69fa6d1f33f78c175 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:27:10.252283 6235 ts_tablet_manager.cc:1398] T 3d024593597a4230b5c25b806843b779 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:27:10.252233 6243 consensus_queue.cc:260] T d702ad56b7e149e69fa6d1f33f78c175 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:10.253782 6236 tablet_bootstrap.cc:492] T d702ad56b7e149e69fa6d1f33f78c175 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:27:10.254040 6243 ts_tablet_manager.cc:1398] T d702ad56b7e149e69fa6d1f33f78c175 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:27:10.255558 6236 tablet_bootstrap.cc:654] T d702ad56b7e149e69fa6d1f33f78c175 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:10.260035 6238 tablet_bootstrap.cc:492] T fb101c0103ec4374b9ca17dea02bf0c4 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:27:10.260213 6238 ts_tablet_manager.cc:1367] T fb101c0103ec4374b9ca17dea02bf0c4 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.021s
I20240502 21:27:10.260891 6238 raft_consensus.cc:348] T fb101c0103ec4374b9ca17dea02bf0c4 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:10.261117 6238 raft_consensus.cc:374] T fb101c0103ec4374b9ca17dea02bf0c4 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:10.261222 6238 raft_consensus.cc:729] T fb101c0103ec4374b9ca17dea02bf0c4 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
I20240502 21:27:10.261724 6238 consensus_queue.cc:260] T fb101c0103ec4374b9ca17dea02bf0c4 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:10.265882 6238 ts_tablet_manager.cc:1398] T fb101c0103ec4374b9ca17dea02bf0c4 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 21:27:10.266696 6236 tablet_bootstrap.cc:492] T d702ad56b7e149e69fa6d1f33f78c175 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:27:10.266868 6236 ts_tablet_manager.cc:1367] T d702ad56b7e149e69fa6d1f33f78c175 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.001s
I20240502 21:27:10.267526 6236 raft_consensus.cc:348] T d702ad56b7e149e69fa6d1f33f78c175 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:10.267745 6236 raft_consensus.cc:374] T d702ad56b7e149e69fa6d1f33f78c175 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:10.267836 6236 raft_consensus.cc:729] T d702ad56b7e149e69fa6d1f33f78c175 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:27:10.268047 6236 consensus_queue.cc:260] T d702ad56b7e149e69fa6d1f33f78c175 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:10.268586 6236 ts_tablet_manager.cc:1398] T d702ad56b7e149e69fa6d1f33f78c175 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240502 21:27:10.269337 31860 consensus_peers.cc:485] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:27:10.270143 6236 tablet_bootstrap.cc:492] T fb101c0103ec4374b9ca17dea02bf0c4 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:27:10.271589 6236 tablet_bootstrap.cc:654] T fb101c0103ec4374b9ca17dea02bf0c4 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:10.278028 6236 tablet_bootstrap.cc:492] T fb101c0103ec4374b9ca17dea02bf0c4 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:27:10.278189 6236 ts_tablet_manager.cc:1367] T fb101c0103ec4374b9ca17dea02bf0c4 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 21:27:10.278868 6236 raft_consensus.cc:348] T fb101c0103ec4374b9ca17dea02bf0c4 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:10.279081 6236 raft_consensus.cc:374] T fb101c0103ec4374b9ca17dea02bf0c4 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:10.279191 6236 raft_consensus.cc:729] T fb101c0103ec4374b9ca17dea02bf0c4 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:27:10.279446 6236 consensus_queue.cc:260] T fb101c0103ec4374b9ca17dea02bf0c4 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:10.280038 6236 ts_tablet_manager.cc:1398] T fb101c0103ec4374b9ca17dea02bf0c4 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:10.291973 6004 raft_consensus.cc:482] T fb101c0103ec4374b9ca17dea02bf0c4 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:10.292078 6004 raft_consensus.cc:504] T fb101c0103ec4374b9ca17dea02bf0c4 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:10.292519 6004 leader_election.cc:288] T fb101c0103ec4374b9ca17dea02bf0c4 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:27:10.292865 2224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb101c0103ec4374b9ca17dea02bf0c4" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:27:10.292994 1831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb101c0103ec4374b9ca17dea02bf0c4" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:27:10.293068 2224 raft_consensus.cc:2454] T fb101c0103ec4374b9ca17dea02bf0c4 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:27:10.293206 1831 raft_consensus.cc:2454] T fb101c0103ec4374b9ca17dea02bf0c4 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 0.
I20240502 21:27:10.293394 31732 leader_election.cc:302] T fb101c0103ec4374b9ca17dea02bf0c4 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:27:10.293622 6004 raft_consensus.cc:2781] T fb101c0103ec4374b9ca17dea02bf0c4 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:10.293695 6004 raft_consensus.cc:482] T fb101c0103ec4374b9ca17dea02bf0c4 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:10.293752 6004 raft_consensus.cc:3037] T fb101c0103ec4374b9ca17dea02bf0c4 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:10.295574 6004 raft_consensus.cc:504] T fb101c0103ec4374b9ca17dea02bf0c4 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:10.296147 6004 leader_election.cc:288] T fb101c0103ec4374b9ca17dea02bf0c4 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Requested vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:27:10.296584 2224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb101c0103ec4374b9ca17dea02bf0c4" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:27:10.296759 2224 raft_consensus.cc:3037] T fb101c0103ec4374b9ca17dea02bf0c4 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:10.298416 2224 raft_consensus.cc:2454] T fb101c0103ec4374b9ca17dea02bf0c4 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:27:10.299237 1831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb101c0103ec4374b9ca17dea02bf0c4" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:27:10.299451 1831 raft_consensus.cc:3037] T fb101c0103ec4374b9ca17dea02bf0c4 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:27:10.301172 31600 consensus_peers.cc:462] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:10.301702 1831 raft_consensus.cc:2454] T fb101c0103ec4374b9ca17dea02bf0c4 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:27:10.302016 31732 leader_election.cc:302] T fb101c0103ec4374b9ca17dea02bf0c4 P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:27:10.302232 6004 raft_consensus.cc:2781] T fb101c0103ec4374b9ca17dea02bf0c4 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:10.302574 6004 raft_consensus.cc:686] T fb101c0103ec4374b9ca17dea02bf0c4 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 LEADER]: Becoming Leader. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Running, Role: LEADER
I20240502 21:27:10.303135 6004 consensus_queue.cc:237] T fb101c0103ec4374b9ca17dea02bf0c4 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:10.304294 31393 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:10.304869 31393 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:10.307799 31393 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet ee39947f94214af8ac16d650c467363e
I20240502 21:27:10.308477 1814 tablet_service.cc:1514] Processing DeleteTablet for tablet ee39947f94214af8ac16d650c467363e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:10 UTC) from {username='slave'} at 127.0.0.1:47510
I20240502 21:27:10.308648 31906 tablet_service.cc:1514] Processing DeleteTablet for tablet ee39947f94214af8ac16d650c467363e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:10 UTC) from {username='slave'} at 127.0.0.1:47824
I20240502 21:27:10.308528 31394 catalog_manager.cc:5484] T fb101c0103ec4374b9ca17dea02bf0c4 P 08d7a0680d244b55b93d17f9ab57b84c reported cstate change: term changed from 0 to 1, leader changed from <none> to 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131). New cstate: current_term: 1 leader_uuid: "08d7a0680d244b55b93d17f9ab57b84c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:10.308872 31647 tablet_service.cc:1514] Processing DeleteTablet for tablet ee39947f94214af8ac16d650c467363e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:10 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:27:10.309916 6252 tablet_replica.cc:330] T ee39947f94214af8ac16d650c467363e P 1e04157b98dd4fad98858c290b73c3db: stopping tablet replica
I20240502 21:27:10.310181 6252 raft_consensus.cc:2229] T ee39947f94214af8ac16d650c467363e P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:10.310451 6252 raft_consensus.cc:2258] T ee39947f94214af8ac16d650c467363e P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:10.311609 6252 ts_tablet_manager.cc:1875] T ee39947f94214af8ac16d650c467363e P 1e04157b98dd4fad98858c290b73c3db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:10.316850 6253 tablet_replica.cc:330] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1: stopping tablet replica
I20240502 21:27:10.317108 6253 raft_consensus.cc:2229] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:10.317127 6252 ts_tablet_manager.cc:1888] T ee39947f94214af8ac16d650c467363e P 1e04157b98dd4fad98858c290b73c3db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:10.317230 6252 log.cc:1196] T ee39947f94214af8ac16d650c467363e P 1e04157b98dd4fad98858c290b73c3db: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal/wals/ee39947f94214af8ac16d650c467363e
I20240502 21:27:10.317569 6253 raft_consensus.cc:2258] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:10.317741 6252 ts_tablet_manager.cc:1909] T ee39947f94214af8ac16d650c467363e P 1e04157b98dd4fad98858c290b73c3db: Deleting consensus metadata
I20240502 21:27:10.318500 6253 ts_tablet_manager.cc:1875] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:10.318724 31381 catalog_manager.cc:4830] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): tablet ee39947f94214af8ac16d650c467363e (table rebalancer_dynamic_table_new_name [id=9fc9a8dd5172486398d9d08dda61f943]) successfully deleted
I20240502 21:27:10.320890 6254 tablet_replica.cc:330] T ee39947f94214af8ac16d650c467363e P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:27:10.321090 6254 raft_consensus.cc:2229] T ee39947f94214af8ac16d650c467363e P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:10.321300 6254 raft_consensus.cc:2258] T ee39947f94214af8ac16d650c467363e P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:10.322163 6254 ts_tablet_manager.cc:1875] T ee39947f94214af8ac16d650c467363e P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:10.323666 6253 ts_tablet_manager.cc:1888] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:10.323781 6253 log.cc:1196] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/wals/ee39947f94214af8ac16d650c467363e
I20240502 21:27:10.324116 6253 ts_tablet_manager.cc:1909] T ee39947f94214af8ac16d650c467363e P 2d50f33557944bb480ea32199d9ee4f1: Deleting consensus metadata
I20240502 21:27:10.325233 31382 catalog_manager.cc:4830] TS 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): tablet ee39947f94214af8ac16d650c467363e (table rebalancer_dynamic_table_new_name [id=9fc9a8dd5172486398d9d08dda61f943]) successfully deleted
I20240502 21:27:10.327498 6254 ts_tablet_manager.cc:1888] T ee39947f94214af8ac16d650c467363e P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:10.327585 6254 log.cc:1196] T ee39947f94214af8ac16d650c467363e P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/ee39947f94214af8ac16d650c467363e
I20240502 21:27:10.327901 6254 ts_tablet_manager.cc:1909] T ee39947f94214af8ac16d650c467363e P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:27:10.328598 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet ee39947f94214af8ac16d650c467363e (table rebalancer_dynamic_table_new_name [id=9fc9a8dd5172486398d9d08dda61f943]) successfully deleted
W20240502 21:27:10.370275 1767 consensus_peers.cc:485] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:27:10.402217 31469 consensus_peers.cc:485] T 51551a1967b74094b373cc030e2bf850 P a44f41b0b6644fb6aa29476aed8c964f -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:27:10.409413 31394 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:10.415954 31514 tablet_service.cc:1467] Processing CreateTablet for tablet a225aea2f2b14f05b43975eeef7f45e9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3db7810fc6e14edfa1231780f7951d10]), partition=
I20240502 21:27:10.416107 31647 tablet_service.cc:1467] Processing CreateTablet for tablet a225aea2f2b14f05b43975eeef7f45e9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3db7810fc6e14edfa1231780f7951d10]), partition=
I20240502 21:27:10.416134 2077 tablet_service.cc:1467] Processing CreateTablet for tablet a225aea2f2b14f05b43975eeef7f45e9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3db7810fc6e14edfa1231780f7951d10]), partition=
I20240502 21:27:10.416287 31514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a225aea2f2b14f05b43975eeef7f45e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:10.416503 2077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a225aea2f2b14f05b43975eeef7f45e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:10.416532 31647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a225aea2f2b14f05b43975eeef7f45e9. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 21:27:10.421880 2030 consensus_peers.cc:485] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:27:10.422469 6243 tablet_bootstrap.cc:492] T a225aea2f2b14f05b43975eeef7f45e9 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
W20240502 21:27:10.422825 31860 consensus_peers.cc:485] T 41029f1332c64ba58ee0ee45714ba2a8 P 2d50f33557944bb480ea32199d9ee4f1 -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:27:10.423553 6255 tablet_bootstrap.cc:492] T a225aea2f2b14f05b43975eeef7f45e9 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:27:10.423936 6243 tablet_bootstrap.cc:654] T a225aea2f2b14f05b43975eeef7f45e9 P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:10.425055 6255 tablet_bootstrap.cc:654] T a225aea2f2b14f05b43975eeef7f45e9 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:10.426072 6237 tablet_bootstrap.cc:492] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:27:10.427666 6237 tablet_bootstrap.cc:654] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:10.433140 6255 tablet_bootstrap.cc:492] T a225aea2f2b14f05b43975eeef7f45e9 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:27:10.433318 6255 ts_tablet_manager.cc:1367] T a225aea2f2b14f05b43975eeef7f45e9 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20240502 21:27:10.433849 6237 tablet_bootstrap.cc:492] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:27:10.433876 6255 raft_consensus.cc:348] T a225aea2f2b14f05b43975eeef7f45e9 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:10.433984 6237 ts_tablet_manager.cc:1367] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.002s
I20240502 21:27:10.434022 6255 raft_consensus.cc:374] T a225aea2f2b14f05b43975eeef7f45e9 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:10.434082 6255 raft_consensus.cc:729] T a225aea2f2b14f05b43975eeef7f45e9 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:27:10.434234 6255 consensus_queue.cc:260] T a225aea2f2b14f05b43975eeef7f45e9 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:10.434667 6255 ts_tablet_manager.cc:1398] T a225aea2f2b14f05b43975eeef7f45e9 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:27:10.434639 6237 raft_consensus.cc:348] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:10.434791 6237 raft_consensus.cc:374] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:10.434834 6237 raft_consensus.cc:729] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:27:10.435003 6237 consensus_queue.cc:260] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:10.435568 6237 ts_tablet_manager.cc:1398] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:27:10.441411 6243 tablet_bootstrap.cc:492] T a225aea2f2b14f05b43975eeef7f45e9 P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:27:10.441797 6243 ts_tablet_manager.cc:1367] T a225aea2f2b14f05b43975eeef7f45e9 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.006s
I20240502 21:27:10.442505 6243 raft_consensus.cc:348] T a225aea2f2b14f05b43975eeef7f45e9 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:10.442718 6243 raft_consensus.cc:374] T a225aea2f2b14f05b43975eeef7f45e9 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:10.442795 6243 raft_consensus.cc:729] T a225aea2f2b14f05b43975eeef7f45e9 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:27:10.442973 6243 consensus_queue.cc:260] T a225aea2f2b14f05b43975eeef7f45e9 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:10.443413 6243 ts_tablet_manager.cc:1398] T a225aea2f2b14f05b43975eeef7f45e9 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:27:10.448309 5661 raft_consensus.cc:482] T 3d024593597a4230b5c25b806843b779 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:10.448524 5661 raft_consensus.cc:504] T 3d024593597a4230b5c25b806843b779 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:10.449168 5661 leader_election.cc:288] T 3d024593597a4230b5c25b806843b779 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:27:10.449569 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d024593597a4230b5c25b806843b779" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:27:10.449769 31797 raft_consensus.cc:2454] T 3d024593597a4230b5c25b806843b779 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:27:10.449928 31923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d024593597a4230b5c25b806843b779" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:27:10.450134 31923 raft_consensus.cc:2454] T 3d024593597a4230b5c25b806843b779 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:27:10.450589 2163 leader_election.cc:302] T 3d024593597a4230b5c25b806843b779 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, 2d50f33557944bb480ea32199d9ee4f1, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:27:10.450824 5661 raft_consensus.cc:2781] T 3d024593597a4230b5c25b806843b779 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:10.450925 5661 raft_consensus.cc:482] T 3d024593597a4230b5c25b806843b779 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:10.451026 5661 raft_consensus.cc:3037] T 3d024593597a4230b5c25b806843b779 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:10.453096 5661 raft_consensus.cc:504] T 3d024593597a4230b5c25b806843b779 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:10.453620 5661 leader_election.cc:288] T 3d024593597a4230b5c25b806843b779 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Requested vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:27:10.453964 31797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d024593597a4230b5c25b806843b779" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:27:10.454085 31923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d024593597a4230b5c25b806843b779" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:27:10.454138 31797 raft_consensus.cc:3037] T 3d024593597a4230b5c25b806843b779 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:10.454212 31923 raft_consensus.cc:3037] T 3d024593597a4230b5c25b806843b779 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:10.455965 31797 raft_consensus.cc:2454] T 3d024593597a4230b5c25b806843b779 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 1.
I20240502 21:27:10.456036 31923 raft_consensus.cc:2454] T 3d024593597a4230b5c25b806843b779 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 1.
I20240502 21:27:10.456357 2164 leader_election.cc:302] T 3d024593597a4230b5c25b806843b779 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:27:10.456615 5661 raft_consensus.cc:2781] T 3d024593597a4230b5c25b806843b779 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:10.456784 5661 raft_consensus.cc:686] T 3d024593597a4230b5c25b806843b779 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Becoming Leader. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Running, Role: LEADER
I20240502 21:27:10.457041 5661 consensus_queue.cc:237] T 3d024593597a4230b5c25b806843b779 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:10.460273 31394 catalog_manager.cc:5484] T 3d024593597a4230b5c25b806843b779 P d1406e92aec84ac585304e57bd37a40d reported cstate change: term changed from 0 to 1, leader changed from <none> to d1406e92aec84ac585304e57bd37a40d (127.24.226.136). New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } }
W20240502 21:27:10.464075 31469 consensus_peers.cc:485] T 4d5d4121e8284195b76bfd12f826f942 P a44f41b0b6644fb6aa29476aed8c964f -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:27:10.504688 31600 consensus_peers.cc:485] T b407c2c9b3a14ce3ada52b7856c6ecee P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:27:10.644948 31469 consensus_peers.cc:485] T f13c3b9c4b7c471fa34156b781473ec4 P a44f41b0b6644fb6aa29476aed8c964f -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:27:10.670090 31730 consensus_peers.cc:485] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 08d7a0680d244b55b93d17f9ab57b84c -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:27:10.670467 31600 consensus_peers.cc:485] T 69432f98b88740adb4fcbc112298cf20 P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:27:10.679530 31600 consensus_peers.cc:462] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:10.680922 5945 raft_consensus.cc:482] T d702ad56b7e149e69fa6d1f33f78c175 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:10.681061 5945 raft_consensus.cc:504] T d702ad56b7e149e69fa6d1f33f78c175 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:10.681692 5945 leader_election.cc:288] T d702ad56b7e149e69fa6d1f33f78c175 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:27:10.681957 31533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d702ad56b7e149e69fa6d1f33f78c175" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:27:10.682152 31533 raft_consensus.cc:2454] T d702ad56b7e149e69fa6d1f33f78c175 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 0.
I20240502 21:27:10.682135 2224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d702ad56b7e149e69fa6d1f33f78c175" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:27:10.682349 2224 raft_consensus.cc:2454] T d702ad56b7e149e69fa6d1f33f78c175 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 0.
I20240502 21:27:10.682528 2033 leader_election.cc:302] T d702ad56b7e149e69fa6d1f33f78c175 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:27:10.682766 5945 raft_consensus.cc:2781] T d702ad56b7e149e69fa6d1f33f78c175 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:10.683070 5945 raft_consensus.cc:482] T d702ad56b7e149e69fa6d1f33f78c175 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:10.683166 5945 raft_consensus.cc:3037] T d702ad56b7e149e69fa6d1f33f78c175 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:10.684988 5945 raft_consensus.cc:504] T d702ad56b7e149e69fa6d1f33f78c175 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:10.685534 5945 leader_election.cc:288] T d702ad56b7e149e69fa6d1f33f78c175 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 election: Requested vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:27:10.685770 31533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d702ad56b7e149e69fa6d1f33f78c175" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:27:10.685936 31533 raft_consensus.cc:3037] T d702ad56b7e149e69fa6d1f33f78c175 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:10.686131 2224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d702ad56b7e149e69fa6d1f33f78c175" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:27:10.686473 2224 raft_consensus.cc:3037] T d702ad56b7e149e69fa6d1f33f78c175 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:10.688705 31533 raft_consensus.cc:2454] T d702ad56b7e149e69fa6d1f33f78c175 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 1.
I20240502 21:27:10.689275 2033 leader_election.cc:302] T d702ad56b7e149e69fa6d1f33f78c175 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:27:10.689469 2224 raft_consensus.cc:2454] T d702ad56b7e149e69fa6d1f33f78c175 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 1.
I20240502 21:27:10.689530 5945 raft_consensus.cc:2781] T d702ad56b7e149e69fa6d1f33f78c175 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election won for term 1
W20240502 21:27:10.689714 31469 consensus_peers.cc:485] T 78f06bc68b954dc4b55b3c8c3c612c2f P a44f41b0b6644fb6aa29476aed8c964f -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:27:10.690026 5945 raft_consensus.cc:686] T d702ad56b7e149e69fa6d1f33f78c175 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 LEADER]: Becoming Leader. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Running, Role: LEADER
I20240502 21:27:10.690517 5945 consensus_queue.cc:237] T d702ad56b7e149e69fa6d1f33f78c175 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:10.693713 31394 catalog_manager.cc:5484] T d702ad56b7e149e69fa6d1f33f78c175 P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135). New cstate: current_term: 1 leader_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } }
W20240502 21:27:10.719219 2161 consensus_peers.cc:485] T a8684b1ca3a642688a232afaf1b28f96 P d1406e92aec84ac585304e57bd37a40d -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:27:10.726521 5417 consensus_queue.cc:1035] T fb101c0103ec4374b9ca17dea02bf0c4 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:10.734557 6004 consensus_queue.cc:1035] T fb101c0103ec4374b9ca17dea02bf0c4 P 08d7a0680d244b55b93d17f9ab57b84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:10.818265 6024 raft_consensus.cc:482] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:10.818400 6024 raft_consensus.cc:504] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:10.818889 6024 leader_election.cc:288] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:27:10.819197 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a225aea2f2b14f05b43975eeef7f45e9" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada" is_pre_election: true
I20240502 21:27:10.819240 2092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a225aea2f2b14f05b43975eeef7f45e9" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:27:10.819416 2092 raft_consensus.cc:2454] T a225aea2f2b14f05b43975eeef7f45e9 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:27:10.819427 31666 raft_consensus.cc:2454] T a225aea2f2b14f05b43975eeef7f45e9 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:27:10.819756 31471 leader_election.cc:302] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:27:10.819957 6024 raft_consensus.cc:2781] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:10.820024 6024 raft_consensus.cc:482] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:10.820081 6024 raft_consensus.cc:3037] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:10.822150 6024 raft_consensus.cc:504] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:10.822553 6024 leader_election.cc:288] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Requested vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:27:10.822942 2092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a225aea2f2b14f05b43975eeef7f45e9" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:27:10.823081 2092 raft_consensus.cc:3037] T a225aea2f2b14f05b43975eeef7f45e9 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:10.825053 2092 raft_consensus.cc:2454] T a225aea2f2b14f05b43975eeef7f45e9 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:27:10.825531 31471 leader_election.cc:302] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:27:10.825748 6024 raft_consensus.cc:2781] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:10.825852 31666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a225aea2f2b14f05b43975eeef7f45e9" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4fac41c4f4959b9b316cb112d0ada"
I20240502 21:27:10.826040 31666 raft_consensus.cc:3037] T a225aea2f2b14f05b43975eeef7f45e9 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:27:10.826148 2030 consensus_peers.cc:485] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:27:10.826197 6024 raft_consensus.cc:686] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:27:10.826464 6024 consensus_queue.cc:237] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:10.829107 31666 raft_consensus.cc:2454] T a225aea2f2b14f05b43975eeef7f45e9 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:27:10.830209 31393 catalog_manager.cc:5484] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 0 to 1, leader changed from <none> to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:10.912369 5661 consensus_queue.cc:1035] T 3d024593597a4230b5c25b806843b779 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:10.916306 6273 consensus_queue.cc:1035] T 3d024593597a4230b5c25b806843b779 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:10.920584 1767 consensus_peers.cc:485] T c977ca41b72d4d45ac7146fbca350824 P 1e04157b98dd4fad98858c290b73c3db -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:27:10.980012 31388 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:10.985334 2092 raft_consensus.cc:1261] T a225aea2f2b14f05b43975eeef7f45e9 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:10.985706 31666 raft_consensus.cc:1261] T a225aea2f2b14f05b43975eeef7f45e9 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Refusing update from remote peer a44f41b0b6644fb6aa29476aed8c964f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:10.985916 6250 consensus_queue.cc:1035] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:10.986366 6271 consensus_queue.cc:1035] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:10.991724 31388 catalog_manager.cc:5576] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569) does not have the latest schema for tablet a225aea2f2b14f05b43975eeef7f45e9 (table rebalancer_dynamic_table [id=3db7810fc6e14edfa1231780f7951d10]). Expected version 1 got 0
I20240502 21:27:10.993299 6281 tablet.cc:1722] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:10.993803 31388 catalog_manager.cc:5576] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645) does not have the latest schema for tablet a225aea2f2b14f05b43975eeef7f45e9 (table rebalancer_dynamic_table [id=3db7810fc6e14edfa1231780f7951d10]). Expected version 1 got 0
I20240502 21:27:10.994267 6281 tablet.cc:1620] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f: MemRowSet was empty: no flush needed.
I20240502 21:27:10.995218 6276 mvcc.cc:205] Tried to move back new op lower bound from 7023350706111488000 to 7023350705467551744. Current Snapshot: MvccSnapshot[applied={T|T < 7023350706111488000}]
I20240502 21:27:10.996541 6283 tablet.cc:1722] T a225aea2f2b14f05b43975eeef7f45e9 P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:10.997128 6283 tablet.cc:1620] T a225aea2f2b14f05b43975eeef7f45e9 P 1ade0354c44c4c37ae480c7a94b11e1b: MemRowSet was empty: no flush needed.
I20240502 21:27:11.002166 31388 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=3db7810fc6e14edfa1231780f7951d10] alter complete (version 1)
I20240502 21:27:11.008780 6282 tablet.cc:1722] T a225aea2f2b14f05b43975eeef7f45e9 P 05f4fac41c4f4959b9b316cb112d0ada: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:11.009429 6282 tablet.cc:1620] T a225aea2f2b14f05b43975eeef7f45e9 P 05f4fac41c4f4959b9b316cb112d0ada: MemRowSet was empty: no flush needed.
I20240502 21:27:11.012105 31394 catalog_manager.cc:5576] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645) does not have the latest schema for tablet a225aea2f2b14f05b43975eeef7f45e9 (table rebalancer_dynamic_table [id=3db7810fc6e14edfa1231780f7951d10]). Expected version 1 got 0
W20240502 21:27:11.031992 2161 consensus_peers.cc:485] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:27:11.044106 31379 catalog_manager.cc:4573] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): CreateTablet RPC failed for tablet eed5a5c41d9a46ce8a712b5b399e0472: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
W20240502 21:27:11.060348 31469 consensus_peers.cc:485] T e4e50c50be16429faef86d47cc324bdb P a44f41b0b6644fb6aa29476aed8c964f -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:27:11.066571 1767 consensus_peers.cc:485] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:27:11.105648 31394 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:11.112254 6281 tablet.cc:1722] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:11.113212 6283 tablet.cc:1722] T a225aea2f2b14f05b43975eeef7f45e9 P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:11.113617 6283 tablet.cc:1620] T a225aea2f2b14f05b43975eeef7f45e9 P 1ade0354c44c4c37ae480c7a94b11e1b: MemRowSet was empty: no flush needed.
I20240502 21:27:11.113976 6282 tablet.cc:1722] T a225aea2f2b14f05b43975eeef7f45e9 P 05f4fac41c4f4959b9b316cb112d0ada: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:11.114365 6282 tablet.cc:1620] T a225aea2f2b14f05b43975eeef7f45e9 P 05f4fac41c4f4959b9b316cb112d0ada: MemRowSet was empty: no flush needed.
I20240502 21:27:11.117909 5945 consensus_queue.cc:1035] T d702ad56b7e149e69fa6d1f33f78c175 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:11.118762 6281 tablet.cc:1620] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f: MemRowSet was empty: no flush needed.
I20240502 21:27:11.123502 31388 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=3db7810fc6e14edfa1231780f7951d10] alter complete (version 2)
I20240502 21:27:11.124517 6118 consensus_queue.cc:1035] T d702ad56b7e149e69fa6d1f33f78c175 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:11.136946 31600 consensus_peers.cc:462] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
W20240502 21:27:11.161329 2030 consensus_peers.cc:485] T 565f30d5539c4b7ea956dc49d0f9a55e P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 21:27:11.187481 31730 consensus_peers.cc:485] T 1df21ee31299490aa8bb95dbd9cfd4d0 P 08d7a0680d244b55b93d17f9ab57b84c -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:27:11.225894 31394 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:11.232604 6281 tablet.cc:1722] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:11.233794 6283 tablet.cc:1722] T a225aea2f2b14f05b43975eeef7f45e9 P 1ade0354c44c4c37ae480c7a94b11e1b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:11.234714 6282 tablet.cc:1722] T a225aea2f2b14f05b43975eeef7f45e9 P 05f4fac41c4f4959b9b316cb112d0ada: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:11.241591 31388 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table_new_name [id=3db7810fc6e14edfa1231780f7951d10] alter complete (version 3)
W20240502 21:27:11.254500 31860 consensus_peers.cc:485] T 88189733d9a14560971dbeb71543073f P 2d50f33557944bb480ea32199d9ee4f1 -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:27:11.261984 31394 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_extra_table_54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 21:27:11.265959 2161 consensus_peers.cc:485] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:27:11.269541 1814 tablet_service.cc:1467] Processing CreateTablet for tablet f59b3b104a4647d59e0ffcfd6288d106 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=e546992c412843fd90d3d775da48a789]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:11.269945 1814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f59b3b104a4647d59e0ffcfd6288d106. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:11.270102 31647 tablet_service.cc:1467] Processing CreateTablet for tablet f59b3b104a4647d59e0ffcfd6288d106 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=e546992c412843fd90d3d775da48a789]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:11.270213 2210 tablet_service.cc:1467] Processing CreateTablet for tablet 77ee047b191e4400ab02040e3fbde8f6 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=e546992c412843fd90d3d775da48a789]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:11.270491 31647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f59b3b104a4647d59e0ffcfd6288d106. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:11.270614 2210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77ee047b191e4400ab02040e3fbde8f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:11.270929 1813 tablet_service.cc:1467] Processing CreateTablet for tablet 77ee047b191e4400ab02040e3fbde8f6 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=e546992c412843fd90d3d775da48a789]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:11.271291 1813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77ee047b191e4400ab02040e3fbde8f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:11.275807 31906 tablet_service.cc:1467] Processing CreateTablet for tablet f59b3b104a4647d59e0ffcfd6288d106 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=e546992c412843fd90d3d775da48a789]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:11.276222 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f59b3b104a4647d59e0ffcfd6288d106. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:11.278303 31774 tablet_service.cc:1467] Processing CreateTablet for tablet 0d0c0636f5534b4389c5b09728c697e7 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=e546992c412843fd90d3d775da48a789]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:11.278779 31774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d0c0636f5534b4389c5b09728c697e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:11.279909 6286 tablet_bootstrap.cc:492] T f59b3b104a4647d59e0ffcfd6288d106 P 05f4fac41c4f4959b9b316cb112d0ada: Bootstrap starting.
I20240502 21:27:11.279858 2077 tablet_service.cc:1467] Processing CreateTablet for tablet 0d0c0636f5534b4389c5b09728c697e7 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=e546992c412843fd90d3d775da48a789]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:11.280301 2077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d0c0636f5534b4389c5b09728c697e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:11.281394 6286 tablet_bootstrap.cc:654] T f59b3b104a4647d59e0ffcfd6288d106 P 05f4fac41c4f4959b9b316cb112d0ada: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:11.284677 31905 tablet_service.cc:1467] Processing CreateTablet for tablet 77ee047b191e4400ab02040e3fbde8f6 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=e546992c412843fd90d3d775da48a789]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:11.285041 31905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77ee047b191e4400ab02040e3fbde8f6. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 21:27:11.286051 31600 consensus_peers.cc:485] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:27:11.286873 6290 tablet_bootstrap.cc:492] T 77ee047b191e4400ab02040e3fbde8f6 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:27:11.288386 6290 tablet_bootstrap.cc:654] T 77ee047b191e4400ab02040e3fbde8f6 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
W20240502 21:27:11.290843 1767 consensus_peers.cc:485] T b67096e38f2e4c42a33e74f0145a04cc P 1e04157b98dd4fad98858c290b73c3db -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:27:11.290975 6291 tablet_bootstrap.cc:492] T 0d0c0636f5534b4389c5b09728c697e7 P 1ade0354c44c4c37ae480c7a94b11e1b: Bootstrap starting.
I20240502 21:27:11.293051 6286 tablet_bootstrap.cc:492] T f59b3b104a4647d59e0ffcfd6288d106 P 05f4fac41c4f4959b9b316cb112d0ada: No bootstrap required, opened a new log
I20240502 21:27:11.293248 6286 ts_tablet_manager.cc:1367] T f59b3b104a4647d59e0ffcfd6288d106 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20240502 21:27:11.294092 6286 raft_consensus.cc:348] T f59b3b104a4647d59e0ffcfd6288d106 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:11.294317 6286 raft_consensus.cc:374] T f59b3b104a4647d59e0ffcfd6288d106 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:11.294422 6290 tablet_bootstrap.cc:492] T 77ee047b191e4400ab02040e3fbde8f6 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:27:11.294436 6286 raft_consensus.cc:729] T f59b3b104a4647d59e0ffcfd6288d106 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Initialized, Role: FOLLOWER
I20240502 21:27:11.294651 6286 consensus_queue.cc:260] T f59b3b104a4647d59e0ffcfd6288d106 P 05f4fac41c4f4959b9b316cb112d0ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:11.294940 6290 ts_tablet_manager.cc:1367] T 77ee047b191e4400ab02040e3fbde8f6 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 21:27:11.295360 6286 ts_tablet_manager.cc:1398] T f59b3b104a4647d59e0ffcfd6288d106 P 05f4fac41c4f4959b9b316cb112d0ada: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:11.295516 31514 tablet_service.cc:1467] Processing CreateTablet for tablet 0d0c0636f5534b4389c5b09728c697e7 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=e546992c412843fd90d3d775da48a789]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:11.295866 31514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d0c0636f5534b4389c5b09728c697e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:11.295898 6290 raft_consensus.cc:348] T 77ee047b191e4400ab02040e3fbde8f6 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:11.297035 6290 raft_consensus.cc:374] T 77ee047b191e4400ab02040e3fbde8f6 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:11.297328 6290 raft_consensus.cc:729] T 77ee047b191e4400ab02040e3fbde8f6 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:27:11.297710 6290 consensus_queue.cc:260] T 77ee047b191e4400ab02040e3fbde8f6 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:11.298599 6290 ts_tablet_manager.cc:1398] T 77ee047b191e4400ab02040e3fbde8f6 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
W20240502 21:27:11.312799 2030 consensus_peers.cc:485] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:27:11.313563 6291 tablet_bootstrap.cc:654] T 0d0c0636f5534b4389c5b09728c697e7 P 1ade0354c44c4c37ae480c7a94b11e1b: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:11.314355 6293 tablet_bootstrap.cc:492] T 0d0c0636f5534b4389c5b09728c697e7 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:27:11.317725 6287 tablet_bootstrap.cc:492] T f59b3b104a4647d59e0ffcfd6288d106 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:27:11.319496 6293 tablet_bootstrap.cc:654] T 0d0c0636f5534b4389c5b09728c697e7 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:11.320343 6289 tablet_bootstrap.cc:492] T f59b3b104a4647d59e0ffcfd6288d106 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:27:11.322182 6289 tablet_bootstrap.cc:654] T f59b3b104a4647d59e0ffcfd6288d106 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:11.322175 6287 tablet_bootstrap.cc:654] T f59b3b104a4647d59e0ffcfd6288d106 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:11.323235 6295 tablet_bootstrap.cc:492] T 0d0c0636f5534b4389c5b09728c697e7 P 08d7a0680d244b55b93d17f9ab57b84c: Bootstrap starting.
I20240502 21:27:11.324716 6295 tablet_bootstrap.cc:654] T 0d0c0636f5534b4389c5b09728c697e7 P 08d7a0680d244b55b93d17f9ab57b84c: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:11.325767 6289 tablet_bootstrap.cc:492] T f59b3b104a4647d59e0ffcfd6288d106 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:27:11.325944 6289 ts_tablet_manager.cc:1367] T f59b3b104a4647d59e0ffcfd6288d106 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 21:27:11.326488 6287 tablet_bootstrap.cc:492] T f59b3b104a4647d59e0ffcfd6288d106 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:27:11.326828 6287 ts_tablet_manager.cc:1367] T f59b3b104a4647d59e0ffcfd6288d106 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240502 21:27:11.326824 6289 raft_consensus.cc:348] T f59b3b104a4647d59e0ffcfd6288d106 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:11.327059 6289 raft_consensus.cc:374] T f59b3b104a4647d59e0ffcfd6288d106 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:11.327183 6289 raft_consensus.cc:729] T f59b3b104a4647d59e0ffcfd6288d106 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:27:11.327447 6289 consensus_queue.cc:260] T f59b3b104a4647d59e0ffcfd6288d106 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:11.327721 6287 raft_consensus.cc:348] T f59b3b104a4647d59e0ffcfd6288d106 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:11.327970 6291 tablet_bootstrap.cc:492] T 0d0c0636f5534b4389c5b09728c697e7 P 1ade0354c44c4c37ae480c7a94b11e1b: No bootstrap required, opened a new log
I20240502 21:27:11.327960 6287 raft_consensus.cc:374] T f59b3b104a4647d59e0ffcfd6288d106 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:11.328058 6289 ts_tablet_manager.cc:1398] T f59b3b104a4647d59e0ffcfd6288d106 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:11.328150 6291 ts_tablet_manager.cc:1367] T 0d0c0636f5534b4389c5b09728c697e7 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.003s
I20240502 21:27:11.328374 6289 tablet_bootstrap.cc:492] T 77ee047b191e4400ab02040e3fbde8f6 P 2d50f33557944bb480ea32199d9ee4f1: Bootstrap starting.
I20240502 21:27:11.328109 6287 raft_consensus.cc:729] T f59b3b104a4647d59e0ffcfd6288d106 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:27:11.330006 6289 tablet_bootstrap.cc:654] T 77ee047b191e4400ab02040e3fbde8f6 P 2d50f33557944bb480ea32199d9ee4f1: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:11.330821 6293 tablet_bootstrap.cc:492] T 0d0c0636f5534b4389c5b09728c697e7 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:27:11.330998 6293 ts_tablet_manager.cc:1367] T 0d0c0636f5534b4389c5b09728c697e7 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240502 21:27:11.331655 6293 raft_consensus.cc:348] T 0d0c0636f5534b4389c5b09728c697e7 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:11.331907 6293 raft_consensus.cc:374] T 0d0c0636f5534b4389c5b09728c697e7 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:11.332016 6293 raft_consensus.cc:729] T 0d0c0636f5534b4389c5b09728c697e7 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:27:11.333518 6293 consensus_queue.cc:260] T 0d0c0636f5534b4389c5b09728c697e7 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:11.335574 6293 ts_tablet_manager.cc:1398] T 0d0c0636f5534b4389c5b09728c697e7 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 21:27:11.335507 6291 raft_consensus.cc:348] T 0d0c0636f5534b4389c5b09728c697e7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:11.329908 6287 consensus_queue.cc:260] T f59b3b104a4647d59e0ffcfd6288d106 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:11.336402 6295 tablet_bootstrap.cc:492] T 0d0c0636f5534b4389c5b09728c697e7 P 08d7a0680d244b55b93d17f9ab57b84c: No bootstrap required, opened a new log
I20240502 21:27:11.338892 6291 raft_consensus.cc:374] T 0d0c0636f5534b4389c5b09728c697e7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:11.339006 6291 raft_consensus.cc:729] T 0d0c0636f5534b4389c5b09728c697e7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Initialized, Role: FOLLOWER
I20240502 21:27:11.339237 6291 consensus_queue.cc:260] T 0d0c0636f5534b4389c5b09728c697e7 P 1ade0354c44c4c37ae480c7a94b11e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:11.339602 6295 ts_tablet_manager.cc:1367] T 0d0c0636f5534b4389c5b09728c697e7 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.001s
I20240502 21:27:11.340210 6287 ts_tablet_manager.cc:1398] T f59b3b104a4647d59e0ffcfd6288d106 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240502 21:27:11.340575 6287 tablet_bootstrap.cc:492] T 77ee047b191e4400ab02040e3fbde8f6 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:27:11.343497 6287 tablet_bootstrap.cc:654] T 77ee047b191e4400ab02040e3fbde8f6 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:11.346796 31388 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:11.346992 31388 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:11.351047 6291 ts_tablet_manager.cc:1398] T 0d0c0636f5534b4389c5b09728c697e7 P 1ade0354c44c4c37ae480c7a94b11e1b: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20240502 21:27:11.352766 31388 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet a225aea2f2b14f05b43975eeef7f45e9
I20240502 21:27:11.353796 2077 tablet_service.cc:1514] Processing DeleteTablet for tablet a225aea2f2b14f05b43975eeef7f45e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:11 UTC) from {username='slave'} at 127.0.0.1:36740
I20240502 21:27:11.354462 6303 tablet_replica.cc:330] T a225aea2f2b14f05b43975eeef7f45e9 P 1ade0354c44c4c37ae480c7a94b11e1b: stopping tablet replica
W20240502 21:27:11.354693 31600 consensus_peers.cc:485] T 182b3f15c06c4c07ade0a8854d3ecafd P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:27:11.354759 31514 tablet_service.cc:1514] Processing DeleteTablet for tablet a225aea2f2b14f05b43975eeef7f45e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:11 UTC) from {username='slave'} at 127.0.0.1:45194
I20240502 21:27:11.355036 31647 tablet_service.cc:1514] Processing DeleteTablet for tablet a225aea2f2b14f05b43975eeef7f45e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:11 UTC) from {username='slave'} at 127.0.0.1:52130
I20240502 21:27:11.351621 6295 raft_consensus.cc:348] T 0d0c0636f5534b4389c5b09728c697e7 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:11.361217 6295 raft_consensus.cc:374] T 0d0c0636f5534b4389c5b09728c697e7 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:11.361357 6295 raft_consensus.cc:729] T 0d0c0636f5534b4389c5b09728c697e7 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d7a0680d244b55b93d17f9ab57b84c, State: Initialized, Role: FOLLOWER
W20240502 21:27:11.361848 31860 consensus_peers.cc:485] T ae56d18c629a4e5eb50911c89de4227c P 2d50f33557944bb480ea32199d9ee4f1 -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:27:11.361653 6295 consensus_queue.cc:260] T 0d0c0636f5534b4389c5b09728c697e7 P 08d7a0680d244b55b93d17f9ab57b84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:11.363095 6295 ts_tablet_manager.cc:1398] T 0d0c0636f5534b4389c5b09728c697e7 P 08d7a0680d244b55b93d17f9ab57b84c: Time spent starting tablet: real 0.023s user 0.000s sys 0.002s
I20240502 21:27:11.364300 6289 tablet_bootstrap.cc:492] T 77ee047b191e4400ab02040e3fbde8f6 P 2d50f33557944bb480ea32199d9ee4f1: No bootstrap required, opened a new log
I20240502 21:27:11.364468 6289 ts_tablet_manager.cc:1367] T 77ee047b191e4400ab02040e3fbde8f6 P 2d50f33557944bb480ea32199d9ee4f1: Time spent bootstrapping tablet: real 0.036s user 0.012s sys 0.011s
I20240502 21:27:11.365212 6289 raft_consensus.cc:348] T 77ee047b191e4400ab02040e3fbde8f6 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:11.365482 6289 raft_consensus.cc:374] T 77ee047b191e4400ab02040e3fbde8f6 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:11.365607 6289 raft_consensus.cc:729] T 77ee047b191e4400ab02040e3fbde8f6 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d50f33557944bb480ea32199d9ee4f1, State: Initialized, Role: FOLLOWER
I20240502 21:27:11.365895 6289 consensus_queue.cc:260] T 77ee047b191e4400ab02040e3fbde8f6 P 2d50f33557944bb480ea32199d9ee4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:11.366721 6289 ts_tablet_manager.cc:1398] T 77ee047b191e4400ab02040e3fbde8f6 P 2d50f33557944bb480ea32199d9ee4f1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:11.369024 6305 tablet_replica.cc:330] T a225aea2f2b14f05b43975eeef7f45e9 P 05f4fac41c4f4959b9b316cb112d0ada: stopping tablet replica
I20240502 21:27:11.372228 6303 raft_consensus.cc:2229] T a225aea2f2b14f05b43975eeef7f45e9 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:11.372743 6305 raft_consensus.cc:2229] T a225aea2f2b14f05b43975eeef7f45e9 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:11.373376 6303 raft_consensus.cc:2258] T a225aea2f2b14f05b43975eeef7f45e9 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:11.373914 6305 raft_consensus.cc:2258] T a225aea2f2b14f05b43975eeef7f45e9 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:11.375536 6303 ts_tablet_manager.cc:1875] T a225aea2f2b14f05b43975eeef7f45e9 P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:11.376582 6305 ts_tablet_manager.cc:1875] T a225aea2f2b14f05b43975eeef7f45e9 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:11.377532 6287 tablet_bootstrap.cc:492] T 77ee047b191e4400ab02040e3fbde8f6 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:27:11.377705 6287 ts_tablet_manager.cc:1367] T 77ee047b191e4400ab02040e3fbde8f6 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.037s user 0.006s sys 0.006s
I20240502 21:27:11.378414 6287 raft_consensus.cc:348] T 77ee047b191e4400ab02040e3fbde8f6 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:11.378620 6287 raft_consensus.cc:374] T 77ee047b191e4400ab02040e3fbde8f6 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:11.378688 6287 raft_consensus.cc:729] T 77ee047b191e4400ab02040e3fbde8f6 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:27:11.378899 6287 consensus_queue.cc:260] T 77ee047b191e4400ab02040e3fbde8f6 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:11.379412 6287 ts_tablet_manager.cc:1398] T 77ee047b191e4400ab02040e3fbde8f6 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20240502 21:27:11.383495 31860 consensus_peers.cc:485] T 62f083e9641042c990efcfb1c4104de5 P 2d50f33557944bb480ea32199d9ee4f1 -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:27:11.384303 6304 tablet_replica.cc:330] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:27:11.385071 6304 raft_consensus.cc:2229] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:11.385905 6303 ts_tablet_manager.cc:1888] T a225aea2f2b14f05b43975eeef7f45e9 P 1ade0354c44c4c37ae480c7a94b11e1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:11.386080 6303 log.cc:1196] T a225aea2f2b14f05b43975eeef7f45e9 P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal/wals/a225aea2f2b14f05b43975eeef7f45e9
I20240502 21:27:11.386291 6304 raft_consensus.cc:2258] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:11.386679 6303 ts_tablet_manager.cc:1909] T a225aea2f2b14f05b43975eeef7f45e9 P 1ade0354c44c4c37ae480c7a94b11e1b: Deleting consensus metadata
I20240502 21:27:11.387171 6304 ts_tablet_manager.cc:1875] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:11.387697 31381 catalog_manager.cc:4830] TS 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569): tablet a225aea2f2b14f05b43975eeef7f45e9 (table rebalancer_dynamic_table_new_name [id=3db7810fc6e14edfa1231780f7951d10]) successfully deleted
I20240502 21:27:11.390036 6305 ts_tablet_manager.cc:1888] T a225aea2f2b14f05b43975eeef7f45e9 P 05f4fac41c4f4959b9b316cb112d0ada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:11.390167 6305 log.cc:1196] T a225aea2f2b14f05b43975eeef7f45e9 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/wals/a225aea2f2b14f05b43975eeef7f45e9
I20240502 21:27:11.390491 6305 ts_tablet_manager.cc:1909] T a225aea2f2b14f05b43975eeef7f45e9 P 05f4fac41c4f4959b9b316cb112d0ada: Deleting consensus metadata
I20240502 21:27:11.391788 31382 catalog_manager.cc:4830] TS 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): tablet a225aea2f2b14f05b43975eeef7f45e9 (table rebalancer_dynamic_table_new_name [id=3db7810fc6e14edfa1231780f7951d10]) successfully deleted
I20240502 21:27:11.394043 6304 ts_tablet_manager.cc:1888] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:11.394145 6304 log.cc:1196] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/a225aea2f2b14f05b43975eeef7f45e9
I20240502 21:27:11.394462 6304 ts_tablet_manager.cc:1909] T a225aea2f2b14f05b43975eeef7f45e9 P a44f41b0b6644fb6aa29476aed8c964f: Deleting consensus metadata
I20240502 21:27:11.395447 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet a225aea2f2b14f05b43975eeef7f45e9 (table rebalancer_dynamic_table_new_name [id=3db7810fc6e14edfa1231780f7951d10]) successfully deleted
W20240502 21:27:11.437147 31469 consensus_peers.cc:485] T 3e6657723fe84e749dea768de44638e7 P a44f41b0b6644fb6aa29476aed8c964f -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:27:11.447530 31469 consensus_peers.cc:485] T 58be3e32928043e1af387efd99e43b63 P a44f41b0b6644fb6aa29476aed8c964f -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:27:11.458061 31394 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50488:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 21:27:11.460608 6232 raft_consensus.cc:482] T f59b3b104a4647d59e0ffcfd6288d106 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:11.460762 6232 raft_consensus.cc:504] T f59b3b104a4647d59e0ffcfd6288d106 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:11.461498 6232 leader_election.cc:288] T f59b3b104a4647d59e0ffcfd6288d106 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:27:11.463196 1831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f59b3b104a4647d59e0ffcfd6288d106" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:27:11.463244 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f59b3b104a4647d59e0ffcfd6288d106" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:27:11.463439 1831 raft_consensus.cc:2454] T f59b3b104a4647d59e0ffcfd6288d106 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:27:11.463464 31925 raft_consensus.cc:2454] T f59b3b104a4647d59e0ffcfd6288d106 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 0.
I20240502 21:27:11.463781 31602 leader_election.cc:302] T f59b3b104a4647d59e0ffcfd6288d106 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 1e04157b98dd4fad98858c290b73c3db; no voters:
I20240502 21:27:11.464021 6232 raft_consensus.cc:2781] T f59b3b104a4647d59e0ffcfd6288d106 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:11.464129 6232 raft_consensus.cc:482] T f59b3b104a4647d59e0ffcfd6288d106 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:11.464218 6232 raft_consensus.cc:3037] T f59b3b104a4647d59e0ffcfd6288d106 P 05f4fac41c4f4959b9b316cb112d0ada [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:11.466527 6232 raft_consensus.cc:504] T f59b3b104a4647d59e0ffcfd6288d106 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:11.468173 31514 tablet_service.cc:1467] Processing CreateTablet for tablet 67c42faaef0c4017a0e4ea1e02c75b62 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9c0d110fe13a41bdbd92442c259f10f4]), partition=
I20240502 21:27:11.468287 2210 tablet_service.cc:1467] Processing CreateTablet for tablet 67c42faaef0c4017a0e4ea1e02c75b62 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9c0d110fe13a41bdbd92442c259f10f4]), partition=
I20240502 21:27:11.468573 31514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67c42faaef0c4017a0e4ea1e02c75b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:11.468570 1814 tablet_service.cc:1467] Processing CreateTablet for tablet 67c42faaef0c4017a0e4ea1e02c75b62 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9c0d110fe13a41bdbd92442c259f10f4]), partition=
I20240502 21:27:11.468668 2210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67c42faaef0c4017a0e4ea1e02c75b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:11.468892 1814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67c42faaef0c4017a0e4ea1e02c75b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:11.469899 1831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f59b3b104a4647d59e0ffcfd6288d106" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:27:11.470124 1831 raft_consensus.cc:3037] T f59b3b104a4647d59e0ffcfd6288d106 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:11.472863 1831 raft_consensus.cc:2454] T f59b3b104a4647d59e0ffcfd6288d106 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:27:11.473233 31602 leader_election.cc:302] T f59b3b104a4647d59e0ffcfd6288d106 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4fac41c4f4959b9b316cb112d0ada, 1e04157b98dd4fad98858c290b73c3db; no voters:
I20240502 21:27:11.473476 6072 raft_consensus.cc:2781] T f59b3b104a4647d59e0ffcfd6288d106 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:11.474239 6232 leader_election.cc:288] T f59b3b104a4647d59e0ffcfd6288d106 P 05f4fac41c4f4959b9b316cb112d0ada [CANDIDATE]: Term 1 election: Requested vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:27:11.474699 6290 tablet_bootstrap.cc:492] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d: Bootstrap starting.
I20240502 21:27:11.476313 6293 tablet_bootstrap.cc:492] T 67c42faaef0c4017a0e4ea1e02c75b62 P a44f41b0b6644fb6aa29476aed8c964f: Bootstrap starting.
I20240502 21:27:11.476413 6290 tablet_bootstrap.cc:654] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:11.477495 6293 tablet_bootstrap.cc:654] T 67c42faaef0c4017a0e4ea1e02c75b62 P a44f41b0b6644fb6aa29476aed8c964f: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:11.478889 6293 tablet_bootstrap.cc:492] T 67c42faaef0c4017a0e4ea1e02c75b62 P a44f41b0b6644fb6aa29476aed8c964f: No bootstrap required, opened a new log
I20240502 21:27:11.479003 6293 ts_tablet_manager.cc:1367] T 67c42faaef0c4017a0e4ea1e02c75b62 P a44f41b0b6644fb6aa29476aed8c964f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:11.478911 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f59b3b104a4647d59e0ffcfd6288d106" candidate_uuid: "05f4fac41c4f4959b9b316cb112d0ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:27:11.479163 31925 raft_consensus.cc:3037] T f59b3b104a4647d59e0ffcfd6288d106 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:11.479526 6293 raft_consensus.cc:348] T 67c42faaef0c4017a0e4ea1e02c75b62 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:11.479652 6293 raft_consensus.cc:374] T 67c42faaef0c4017a0e4ea1e02c75b62 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:11.479698 6293 raft_consensus.cc:729] T 67c42faaef0c4017a0e4ea1e02c75b62 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Initialized, Role: FOLLOWER
I20240502 21:27:11.479871 6293 consensus_queue.cc:260] T 67c42faaef0c4017a0e4ea1e02c75b62 P a44f41b0b6644fb6aa29476aed8c964f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:11.482631 6290 tablet_bootstrap.cc:492] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d: No bootstrap required, opened a new log
I20240502 21:27:11.483115 6290 ts_tablet_manager.cc:1367] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20240502 21:27:11.483198 6293 ts_tablet_manager.cc:1398] T 67c42faaef0c4017a0e4ea1e02c75b62 P a44f41b0b6644fb6aa29476aed8c964f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:11.483358 6072 raft_consensus.cc:686] T f59b3b104a4647d59e0ffcfd6288d106 P 05f4fac41c4f4959b9b316cb112d0ada [term 1 LEADER]: Becoming Leader. State: Replica: 05f4fac41c4f4959b9b316cb112d0ada, State: Running, Role: LEADER
I20240502 21:27:11.483647 6072 consensus_queue.cc:237] T f59b3b104a4647d59e0ffcfd6288d106 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:11.483855 6290 raft_consensus.cc:348] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:11.484158 6250 raft_consensus.cc:482] T 0d0c0636f5534b4389c5b09728c697e7 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:11.484279 6250 raft_consensus.cc:504] T 0d0c0636f5534b4389c5b09728c697e7 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:11.484894 6250 leader_election.cc:288] T 0d0c0636f5534b4389c5b09728c697e7 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:27:11.482405 31925 raft_consensus.cc:2454] T f59b3b104a4647d59e0ffcfd6288d106 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4fac41c4f4959b9b316cb112d0ada in term 1.
I20240502 21:27:11.485077 6290 raft_consensus.cc:374] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:11.485249 6290 raft_consensus.cc:729] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Initialized, Role: FOLLOWER
I20240502 21:27:11.485618 6290 consensus_queue.cc:260] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:11.485792 31793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d0c0636f5534b4389c5b09728c697e7" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c" is_pre_election: true
I20240502 21:27:11.486029 31793 raft_consensus.cc:2454] T 0d0c0636f5534b4389c5b09728c697e7 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:27:11.486269 6290 ts_tablet_manager.cc:1398] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:27:11.486629 31471 leader_election.cc:302] T 0d0c0636f5534b4389c5b09728c697e7 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:27:11.487043 6250 raft_consensus.cc:2781] T 0d0c0636f5534b4389c5b09728c697e7 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:11.487154 6250 raft_consensus.cc:482] T 0d0c0636f5534b4389c5b09728c697e7 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:11.487277 6250 raft_consensus.cc:3037] T 0d0c0636f5534b4389c5b09728c697e7 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:11.487524 6287 tablet_bootstrap.cc:492] T 67c42faaef0c4017a0e4ea1e02c75b62 P 1e04157b98dd4fad98858c290b73c3db: Bootstrap starting.
I20240502 21:27:11.488591 6287 tablet_bootstrap.cc:654] T 67c42faaef0c4017a0e4ea1e02c75b62 P 1e04157b98dd4fad98858c290b73c3db: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:11.490195 6250 raft_consensus.cc:504] T 0d0c0636f5534b4389c5b09728c697e7 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
W20240502 21:27:11.494876 31730 consensus_peers.cc:485] T 9bc57b8a0388416290997df544e5cc0b P 08d7a0680d244b55b93d17f9ab57b84c -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:27:11.495441 6250 leader_election.cc:288] T 0d0c0636f5534b4389c5b09728c697e7 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Requested vote from peers 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
I20240502 21:27:11.495704 2092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d0c0636f5534b4389c5b09728c697e7" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b"
I20240502 21:27:11.495771 31793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d0c0636f5534b4389c5b09728c697e7" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d7a0680d244b55b93d17f9ab57b84c"
I20240502 21:27:11.495796 2093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d0c0636f5534b4389c5b09728c697e7" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
I20240502 21:27:11.495929 31793 raft_consensus.cc:3037] T 0d0c0636f5534b4389c5b09728c697e7 P 08d7a0680d244b55b93d17f9ab57b84c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:11.495939 2092 raft_consensus.cc:3037] T 0d0c0636f5534b4389c5b09728c697e7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:11.498503 2092 raft_consensus.cc:2454] T 0d0c0636f5534b4389c5b09728c697e7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
I20240502 21:27:11.498529 31793 raft_consensus.cc:2454] T 0d0c0636f5534b4389c5b09728c697e7 P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 1.
W20240502 21:27:11.498905 31469 consensus_peers.cc:485] T 759b7529d7d84baab1a4622d34f803bf P a44f41b0b6644fb6aa29476aed8c964f -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:27:11.499430 31471 leader_election.cc:302] T 0d0c0636f5534b4389c5b09728c697e7 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:27:11.499675 6250 raft_consensus.cc:2781] T 0d0c0636f5534b4389c5b09728c697e7 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:11.500118 6250 raft_consensus.cc:686] T 0d0c0636f5534b4389c5b09728c697e7 P a44f41b0b6644fb6aa29476aed8c964f [term 1 LEADER]: Becoming Leader. State: Replica: a44f41b0b6644fb6aa29476aed8c964f, State: Running, Role: LEADER
I20240502 21:27:11.500413 6250 consensus_queue.cc:237] T 0d0c0636f5534b4389c5b09728c697e7 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:11.502424 31388 catalog_manager.cc:5484] T f59b3b104a4647d59e0ffcfd6288d106 P 05f4fac41c4f4959b9b316cb112d0ada reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130). New cstate: current_term: 1 leader_uuid: "05f4fac41c4f4959b9b316cb112d0ada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:11.504664 6287 tablet_bootstrap.cc:492] T 67c42faaef0c4017a0e4ea1e02c75b62 P 1e04157b98dd4fad98858c290b73c3db: No bootstrap required, opened a new log
I20240502 21:27:11.504860 6287 ts_tablet_manager.cc:1367] T 67c42faaef0c4017a0e4ea1e02c75b62 P 1e04157b98dd4fad98858c290b73c3db: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.010s
I20240502 21:27:11.505647 6287 raft_consensus.cc:348] T 67c42faaef0c4017a0e4ea1e02c75b62 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:11.505909 6287 raft_consensus.cc:374] T 67c42faaef0c4017a0e4ea1e02c75b62 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:11.506052 6287 raft_consensus.cc:729] T 67c42faaef0c4017a0e4ea1e02c75b62 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04157b98dd4fad98858c290b73c3db, State: Initialized, Role: FOLLOWER
I20240502 21:27:11.506347 6287 consensus_queue.cc:260] T 67c42faaef0c4017a0e4ea1e02c75b62 P 1e04157b98dd4fad98858c290b73c3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:11.507046 6287 ts_tablet_manager.cc:1398] T 67c42faaef0c4017a0e4ea1e02c75b62 P 1e04157b98dd4fad98858c290b73c3db: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240502 21:27:11.509202 2161 consensus_peers.cc:485] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:27:11.512292 31394 catalog_manager.cc:5484] T 0d0c0636f5534b4389c5b09728c697e7 P a44f41b0b6644fb6aa29476aed8c964f reported cstate change: term changed from 0 to 1, leader changed from <none> to a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129). New cstate: current_term: 1 leader_uuid: "a44f41b0b6644fb6aa29476aed8c964f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: HEALTHY } } }
W20240502 21:27:11.516606 31469 consensus_peers.cc:485] T 55fc18e4cc98446ca50c1dd2b6a9a89d P a44f41b0b6644fb6aa29476aed8c964f -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:27:11.527288 31600 consensus_peers.cc:485] T 0b8d119ee8474a40ac08df61df5b8423 P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:27:11.580775 2161 consensus_peers.cc:485] T a8328088928f4de19f7cf20189382d91 P d1406e92aec84ac585304e57bd37a40d -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:27:11.589723 6234 raft_consensus.cc:482] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:11.589877 6234 raft_consensus.cc:504] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:11.590348 6250 raft_consensus.cc:482] T 67c42faaef0c4017a0e4ea1e02c75b62 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:11.590513 6250 raft_consensus.cc:504] T 67c42faaef0c4017a0e4ea1e02c75b62 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:11.590677 6234 leader_election.cc:288] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:27:11.591168 6250 leader_election.cc:288] T 67c42faaef0c4017a0e4ea1e02c75b62 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:27:11.591303 1831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c42faaef0c4017a0e4ea1e02c75b62" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:27:11.591543 1831 raft_consensus.cc:2454] T 67c42faaef0c4017a0e4ea1e02c75b62 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:27:11.591616 1834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c42faaef0c4017a0e4ea1e02c75b62" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:27:11.591825 1834 raft_consensus.cc:2454] T 67c42faaef0c4017a0e4ea1e02c75b62 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a44f41b0b6644fb6aa29476aed8c964f in term 0.
I20240502 21:27:11.592114 2163 leader_election.cc:302] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:27:11.592275 31471 leader_election.cc:302] T 67c42faaef0c4017a0e4ea1e02c75b62 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, a44f41b0b6644fb6aa29476aed8c964f; no voters:
I20240502 21:27:11.592070 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c42faaef0c4017a0e4ea1e02c75b62" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f" is_pre_election: true
I20240502 21:27:11.592404 6234 raft_consensus.cc:2781] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:11.592543 6234 raft_consensus.cc:482] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:11.592528 31535 raft_consensus.cc:2454] T 67c42faaef0c4017a0e4ea1e02c75b62 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:27:11.592638 6234 raft_consensus.cc:3037] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:11.592823 6250 raft_consensus.cc:2781] T 67c42faaef0c4017a0e4ea1e02c75b62 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:11.592993 6250 raft_consensus.cc:482] T 67c42faaef0c4017a0e4ea1e02c75b62 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:11.593150 6250 raft_consensus.cc:3037] T 67c42faaef0c4017a0e4ea1e02c75b62 P a44f41b0b6644fb6aa29476aed8c964f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:11.593089 2225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c42faaef0c4017a0e4ea1e02c75b62" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:27:11.595000 6234 raft_consensus.cc:504] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:11.595412 2225 raft_consensus.cc:2379] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a44f41b0b6644fb6aa29476aed8c964f in current term 1: Already voted for candidate d1406e92aec84ac585304e57bd37a40d in this term.
I20240502 21:27:11.595321 6250 raft_consensus.cc:504] T 67c42faaef0c4017a0e4ea1e02c75b62 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:11.595651 6234 leader_election.cc:288] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Requested vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:27:11.596161 6250 leader_election.cc:288] T 67c42faaef0c4017a0e4ea1e02c75b62 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Requested vote from peers d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:27:11.596117 1834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c42faaef0c4017a0e4ea1e02c75b62" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:27:11.596225 31535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c42faaef0c4017a0e4ea1e02c75b62" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44f41b0b6644fb6aa29476aed8c964f"
I20240502 21:27:11.596302 1834 raft_consensus.cc:3037] T 67c42faaef0c4017a0e4ea1e02c75b62 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:11.596493 31535 raft_consensus.cc:2379] T 67c42faaef0c4017a0e4ea1e02c75b62 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d1406e92aec84ac585304e57bd37a40d in current term 1: Already voted for candidate a44f41b0b6644fb6aa29476aed8c964f in this term.
I20240502 21:27:11.596819 1831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c42faaef0c4017a0e4ea1e02c75b62" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:27:11.597936 2225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c42faaef0c4017a0e4ea1e02c75b62" candidate_uuid: "a44f41b0b6644fb6aa29476aed8c964f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:27:11.598189 2225 raft_consensus.cc:2379] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a44f41b0b6644fb6aa29476aed8c964f in current term 1: Already voted for candidate d1406e92aec84ac585304e57bd37a40d in this term.
W20240502 21:27:11.598395 31600 consensus_peers.cc:485] T 925797977edf495ca09aa25988d14fac P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:27:11.599064 1834 raft_consensus.cc:2454] T 67c42faaef0c4017a0e4ea1e02c75b62 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 1.
I20240502 21:27:11.599113 31471 leader_election.cc:302] T 67c42faaef0c4017a0e4ea1e02c75b62 P a44f41b0b6644fb6aa29476aed8c964f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a44f41b0b6644fb6aa29476aed8c964f; no voters: 1e04157b98dd4fad98858c290b73c3db, d1406e92aec84ac585304e57bd37a40d
I20240502 21:27:11.599452 2163 leader_election.cc:302] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db, d1406e92aec84ac585304e57bd37a40d; no voters: a44f41b0b6644fb6aa29476aed8c964f
I20240502 21:27:11.599831 6250 raft_consensus.cc:2726] T 67c42faaef0c4017a0e4ea1e02c75b62 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 21:27:11.600595 6234 raft_consensus.cc:2781] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:11.600729 6234 raft_consensus.cc:686] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Becoming Leader. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Running, Role: LEADER
I20240502 21:27:11.600946 6234 consensus_queue.cc:237] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:11.605090 31389 catalog_manager.cc:5484] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d reported cstate change: term changed from 0 to 1, leader changed from <none> to d1406e92aec84ac585304e57bd37a40d (127.24.226.136). New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:11.617610 6234 raft_consensus.cc:482] T 77ee047b191e4400ab02040e3fbde8f6 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:11.617754 6234 raft_consensus.cc:504] T 77ee047b191e4400ab02040e3fbde8f6 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:11.618427 6234 leader_election.cc:288] T 77ee047b191e4400ab02040e3fbde8f6 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:27:11.618961 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ee047b191e4400ab02040e3fbde8f6" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
I20240502 21:27:11.619194 31925 raft_consensus.cc:2454] T 77ee047b191e4400ab02040e3fbde8f6 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:27:11.619243 1834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ee047b191e4400ab02040e3fbde8f6" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:27:11.619458 1834 raft_consensus.cc:2454] T 77ee047b191e4400ab02040e3fbde8f6 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 0.
I20240502 21:27:11.619628 2164 leader_election.cc:302] T 77ee047b191e4400ab02040e3fbde8f6 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:27:11.619884 6234 raft_consensus.cc:2781] T 77ee047b191e4400ab02040e3fbde8f6 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:11.619984 6234 raft_consensus.cc:482] T 77ee047b191e4400ab02040e3fbde8f6 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:11.620121 6234 raft_consensus.cc:3037] T 77ee047b191e4400ab02040e3fbde8f6 P d1406e92aec84ac585304e57bd37a40d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:11.622826 6234 raft_consensus.cc:504] T 77ee047b191e4400ab02040e3fbde8f6 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:11.624572 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ee047b191e4400ab02040e3fbde8f6" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1"
I20240502 21:27:11.624867 6234 leader_election.cc:288] T 77ee047b191e4400ab02040e3fbde8f6 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Requested vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:27:11.624756 31925 raft_consensus.cc:3037] T 77ee047b191e4400ab02040e3fbde8f6 P 2d50f33557944bb480ea32199d9ee4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:11.625070 1834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ee047b191e4400ab02040e3fbde8f6" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db"
I20240502 21:27:11.625260 1834 raft_consensus.cc:3037] T 77ee047b191e4400ab02040e3fbde8f6 P 1e04157b98dd4fad98858c290b73c3db [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:11.627590 31925 raft_consensus.cc:2454] T 77ee047b191e4400ab02040e3fbde8f6 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 1.
I20240502 21:27:11.627930 2164 leader_election.cc:302] T 77ee047b191e4400ab02040e3fbde8f6 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1, d1406e92aec84ac585304e57bd37a40d; no voters:
I20240502 21:27:11.627760 1834 raft_consensus.cc:2454] T 77ee047b191e4400ab02040e3fbde8f6 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1406e92aec84ac585304e57bd37a40d in term 1.
I20240502 21:27:11.628567 5661 raft_consensus.cc:2781] T 77ee047b191e4400ab02040e3fbde8f6 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:11.628767 5661 raft_consensus.cc:686] T 77ee047b191e4400ab02040e3fbde8f6 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Becoming Leader. State: Replica: d1406e92aec84ac585304e57bd37a40d, State: Running, Role: LEADER
I20240502 21:27:11.629025 5661 consensus_queue.cc:237] T 77ee047b191e4400ab02040e3fbde8f6 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
W20240502 21:27:11.631844 2030 consensus_peers.cc:485] T b1d59ebecac34cf7b33424cc73cff030 P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:27:11.633839 31389 catalog_manager.cc:5484] T 77ee047b191e4400ab02040e3fbde8f6 P d1406e92aec84ac585304e57bd37a40d reported cstate change: term changed from 0 to 1, leader changed from <none> to d1406e92aec84ac585304e57bd37a40d (127.24.226.136). New cstate: current_term: 1 leader_uuid: "d1406e92aec84ac585304e57bd37a40d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } health_report { overall_health: UNKNOWN } } }
W20240502 21:27:11.637507 2030 consensus_peers.cc:485] T 939c34669f6447f9a2a88da769d0f84f P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:27:11.646196 31600 consensus_peers.cc:462] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
W20240502 21:27:11.652194 31730 consensus_peers.cc:485] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:27:11.696123 2161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240502 21:27:11.697394 1767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111) [suppressed 55 similar messages]
W20240502 21:27:11.712291 31469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111) [suppressed 107 similar messages]
W20240502 21:27:11.713089 31469 consensus_peers.cc:485] T 5bc6c7211d5f4f73b2a050b9ec78c6b1 P a44f41b0b6644fb6aa29476aed8c964f -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:27:11.715544 31389 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 21:27:11.721029 1834 raft_consensus.cc:1261] T 67c42faaef0c4017a0e4ea1e02c75b62 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:11.721107 31535 raft_consensus.cc:1261] T 67c42faaef0c4017a0e4ea1e02c75b62 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Refusing update from remote peer d1406e92aec84ac585304e57bd37a40d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:11.721742 6234 consensus_queue.cc:1035] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:11.722186 5661 consensus_queue.cc:1035] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:11.727210 31389 catalog_manager.cc:5576] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039) does not have the latest schema for tablet 67c42faaef0c4017a0e4ea1e02c75b62 (table rebalancer_dynamic_table [id=9c0d110fe13a41bdbd92442c259f10f4]). Expected version 1 got 0
I20240502 21:27:11.727988 6281 tablet.cc:1722] T 67c42faaef0c4017a0e4ea1e02c75b62 P a44f41b0b6644fb6aa29476aed8c964f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:11.728250 6318 tablet.cc:1722] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:11.728758 6318 tablet.cc:1620] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d: MemRowSet was empty: no flush needed.
I20240502 21:27:11.732746 6281 tablet.cc:1620] T 67c42faaef0c4017a0e4ea1e02c75b62 P a44f41b0b6644fb6aa29476aed8c964f: MemRowSet was empty: no flush needed.
I20240502 21:27:11.738725 31389 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=9c0d110fe13a41bdbd92442c259f10f4] alter complete (version 1)
I20240502 21:27:11.743207 6319 tablet.cc:1722] T 67c42faaef0c4017a0e4ea1e02c75b62 P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 21:27:11.743618 6316 mvcc.cc:205] Tried to move back new op lower bound from 7023350709124681728 to 7023350708639285248. Current Snapshot: MvccSnapshot[applied={T|T < 7023350709124681728}]
I20240502 21:27:11.745399 6319 tablet.cc:1620] T 67c42faaef0c4017a0e4ea1e02c75b62 P 1e04157b98dd4fad98858c290b73c3db: MemRowSet was empty: no flush needed.
W20240502 21:27:11.751022 31730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111) [suppressed 43 similar messages]
I20240502 21:27:11.753469 31394 catalog_manager.cc:5576] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863) does not have the latest schema for tablet 67c42faaef0c4017a0e4ea1e02c75b62 (table rebalancer_dynamic_table [id=9c0d110fe13a41bdbd92442c259f10f4]). Expected version 1 got 0
W20240502 21:27:11.784999 31600 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111) [suppressed 85 similar messages]
W20240502 21:27:11.842236 2161 consensus_peers.cc:485] T 84ba90ce8dd24ea2b325b1dba2e4c494 P d1406e92aec84ac585304e57bd37a40d -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:27:11.843833 31394 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 21:27:11.851296 6318 tablet.cc:1722] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:11.851775 6318 tablet.cc:1620] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d: MemRowSet was empty: no flush needed.
I20240502 21:27:11.851769 6281 tablet.cc:1722] T 67c42faaef0c4017a0e4ea1e02c75b62 P a44f41b0b6644fb6aa29476aed8c964f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:11.852146 6281 tablet.cc:1620] T 67c42faaef0c4017a0e4ea1e02c75b62 P a44f41b0b6644fb6aa29476aed8c964f: MemRowSet was empty: no flush needed.
I20240502 21:27:11.864760 6319 tablet.cc:1722] T 67c42faaef0c4017a0e4ea1e02c75b62 P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 21:27:11.865260 6319 tablet.cc:1620] T 67c42faaef0c4017a0e4ea1e02c75b62 P 1e04157b98dd4fad98858c290b73c3db: MemRowSet was empty: no flush needed.
I20240502 21:27:11.862499 31388 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table [id=9c0d110fe13a41bdbd92442c259f10f4] alter complete (version 2)
W20240502 21:27:11.879920 2030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111) [suppressed 65 similar messages]
I20240502 21:27:11.882941 6250 consensus_queue.cc:1035] T 0d0c0636f5534b4389c5b09728c697e7 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:11.890913 6321 consensus_queue.cc:1035] T 0d0c0636f5534b4389c5b09728c697e7 P a44f41b0b6644fb6aa29476aed8c964f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:11.895409 31860 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111) [suppressed 56 similar messages]
I20240502 21:27:11.913955 6306 consensus_queue.cc:1035] T f59b3b104a4647d59e0ffcfd6288d106 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:11.920382 6306 consensus_queue.cc:1035] T f59b3b104a4647d59e0ffcfd6288d106 P 05f4fac41c4f4959b9b316cb112d0ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:11.924966 1767 consensus_peers.cc:485] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:27:11.933156 31379 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 21:27:11.967806 31388 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 21:27:11.973650 6318 tablet.cc:1722] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:11.974274 6319 tablet.cc:1722] T 67c42faaef0c4017a0e4ea1e02c75b62 P 1e04157b98dd4fad98858c290b73c3db: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:11.974483 6281 tablet.cc:1722] T 67c42faaef0c4017a0e4ea1e02c75b62 P a44f41b0b6644fb6aa29476aed8c964f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 21:27:11.982990 31388 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: rebalancer_dynamic_table_new_name [id=9c0d110fe13a41bdbd92442c259f10f4] alter complete (version 3)
I20240502 21:27:12.085744 31389 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:12.085963 31389 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50488:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 21:27:12.089154 31389 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5afb5e5a489743fe980538330e79474e: Sending DeleteTablet for 3 replicas of tablet 67c42faaef0c4017a0e4ea1e02c75b62
I20240502 21:27:12.090116 2210 tablet_service.cc:1514] Processing DeleteTablet for tablet 67c42faaef0c4017a0e4ea1e02c75b62 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:12 UTC) from {username='slave'} at 127.0.0.1:36984
I20240502 21:27:12.090423 31514 tablet_service.cc:1514] Processing DeleteTablet for tablet 67c42faaef0c4017a0e4ea1e02c75b62 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:12 UTC) from {username='slave'} at 127.0.0.1:45194
I20240502 21:27:12.090615 1814 tablet_service.cc:1514] Processing DeleteTablet for tablet 67c42faaef0c4017a0e4ea1e02c75b62 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 21:27:12 UTC) from {username='slave'} at 127.0.0.1:47510
I20240502 21:27:12.091202 6331 tablet_replica.cc:330] T 67c42faaef0c4017a0e4ea1e02c75b62 P a44f41b0b6644fb6aa29476aed8c964f: stopping tablet replica
I20240502 21:27:12.091538 6331 raft_consensus.cc:2229] T 67c42faaef0c4017a0e4ea1e02c75b62 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:12.091753 6331 raft_consensus.cc:2258] T 67c42faaef0c4017a0e4ea1e02c75b62 P a44f41b0b6644fb6aa29476aed8c964f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:12.093248 6331 ts_tablet_manager.cc:1875] T 67c42faaef0c4017a0e4ea1e02c75b62 P a44f41b0b6644fb6aa29476aed8c964f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:12.097981 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 31454
I20240502 21:27:12.099887 6331 ts_tablet_manager.cc:1888] T 67c42faaef0c4017a0e4ea1e02c75b62 P a44f41b0b6644fb6aa29476aed8c964f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:12.100004 6331 log.cc:1196] T 67c42faaef0c4017a0e4ea1e02c75b62 P a44f41b0b6644fb6aa29476aed8c964f: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/wals/67c42faaef0c4017a0e4ea1e02c75b62
I20240502 21:27:12.100419 6331 ts_tablet_manager.cc:1909] T 67c42faaef0c4017a0e4ea1e02c75b62 P a44f41b0b6644fb6aa29476aed8c964f: Deleting consensus metadata
I20240502 21:27:12.102123 6332 tablet_replica.cc:330] T 67c42faaef0c4017a0e4ea1e02c75b62 P 1e04157b98dd4fad98858c290b73c3db: stopping tablet replica
I20240502 21:27:12.102360 6332 raft_consensus.cc:2229] T 67c42faaef0c4017a0e4ea1e02c75b62 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:12.102900 31382 catalog_manager.cc:4830] TS a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): tablet 67c42faaef0c4017a0e4ea1e02c75b62 (table rebalancer_dynamic_table_new_name [id=9c0d110fe13a41bdbd92442c259f10f4]) successfully deleted
I20240502 21:27:12.103119 6330 tablet_replica.cc:330] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d: stopping tablet replica
I20240502 21:27:12.103375 6330 raft_consensus.cc:2229] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:27:12.103788 6330 raft_consensus.cc:2258] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:12.103788 6332 raft_consensus.cc:2258] T 67c42faaef0c4017a0e4ea1e02c75b62 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:12.104624 6330 ts_tablet_manager.cc:1875] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:12.104801 6332 ts_tablet_manager.cc:1875] T 67c42faaef0c4017a0e4ea1e02c75b62 P 1e04157b98dd4fad98858c290b73c3db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 21:27:12.112597 6332 ts_tablet_manager.cc:1888] T 67c42faaef0c4017a0e4ea1e02c75b62 P 1e04157b98dd4fad98858c290b73c3db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:12.112771 6332 log.cc:1196] T 67c42faaef0c4017a0e4ea1e02c75b62 P 1e04157b98dd4fad98858c290b73c3db: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal/wals/67c42faaef0c4017a0e4ea1e02c75b62
I20240502 21:27:12.113339 6332 ts_tablet_manager.cc:1909] T 67c42faaef0c4017a0e4ea1e02c75b62 P 1e04157b98dd4fad98858c290b73c3db: Deleting consensus metadata
I20240502 21:27:12.114451 31381 catalog_manager.cc:4830] TS 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): tablet 67c42faaef0c4017a0e4ea1e02c75b62 (table rebalancer_dynamic_table_new_name [id=9c0d110fe13a41bdbd92442c259f10f4]) successfully deleted
I20240502 21:27:12.114789 31575 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:27:12.115291 6330 ts_tablet_manager.cc:1888] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 21:27:12.115514 6330 log.cc:1196] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d: Deleting WAL directory at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal/wals/67c42faaef0c4017a0e4ea1e02c75b62
I20240502 21:27:12.115918 6330 ts_tablet_manager.cc:1909] T 67c42faaef0c4017a0e4ea1e02c75b62 P d1406e92aec84ac585304e57bd37a40d: Deleting consensus metadata
I20240502 21:27:12.116864 31381 catalog_manager.cc:4830] TS d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907): tablet 67c42faaef0c4017a0e4ea1e02c75b62 (table rebalancer_dynamic_table_new_name [id=9c0d110fe13a41bdbd92442c259f10f4]) successfully deleted
I20240502 21:27:12.152381 6273 consensus_queue.cc:1035] T 77ee047b191e4400ab02040e3fbde8f6 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:12.157838 6273 consensus_queue.cc:1035] T 77ee047b191e4400ab02040e3fbde8f6 P d1406e92aec84ac585304e57bd37a40d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:12.184842 31600 consensus_peers.cc:462] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:12.454713 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 31454
W20240502 21:27:12.500491 1770 connection.cc:535] client connection to 127.24.226.129:41039 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:12.500689 31603 connection.cc:535] client connection to 127.24.226.129:41039 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:12.501040 31602 connection.cc:535] server connection from 127.24.226.129:48937 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:27:12.501278 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 31585
W20240502 21:27:12.501276 2033 connection.cc:535] client connection to 127.24.226.129:41039 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:12.501390 31733 connection.cc:535] client connection to 127.24.226.129:41039 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:12.502573 1770 consensus_peers.cc:485] T e84560e2ba25443cbba6babbbfcaf104 P 1e04157b98dd4fad98858c290b73c3db -> Peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Couldn't send request to peer a44f41b0b6644fb6aa29476aed8c964f. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:12.502697 1770 consensus_peers.cc:485] T 5d045ba2da9e482da25a39a0a2f38ff5 P 1e04157b98dd4fad98858c290b73c3db -> Peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Couldn't send request to peer a44f41b0b6644fb6aa29476aed8c964f. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:12.502835 31733 consensus_peers.cc:485] T 815b1fc784714c8cb4ba76707c345625 P 08d7a0680d244b55b93d17f9ab57b84c -> Peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Couldn't send request to peer a44f41b0b6644fb6aa29476aed8c964f. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:12.502925 31733 consensus_peers.cc:485] T 44277dc39b8a4729b704986edbfb9b84 P 08d7a0680d244b55b93d17f9ab57b84c -> Peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Couldn't send request to peer a44f41b0b6644fb6aa29476aed8c964f. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:12.503000 31733 consensus_peers.cc:485] T a1fd14de204c40d98027ecb329ff79f3 P 08d7a0680d244b55b93d17f9ab57b84c -> Peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Couldn't send request to peer a44f41b0b6644fb6aa29476aed8c964f. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:12.503392 2033 consensus_peers.cc:485] T d702ad56b7e149e69fa6d1f33f78c175 P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Couldn't send request to peer a44f41b0b6644fb6aa29476aed8c964f. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:12.503515 2033 consensus_peers.cc:485] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Couldn't send request to peer a44f41b0b6644fb6aa29476aed8c964f. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:12.503677 31603 consensus_peers.cc:485] T b20c2bb60f6149458a62ee443b8cf3ab P 05f4fac41c4f4959b9b316cb112d0ada -> Peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Couldn't send request to peer a44f41b0b6644fb6aa29476aed8c964f. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:12.503773 31603 consensus_peers.cc:485] T 043f33cb58084c3ebbce4893d1b63fbc P 05f4fac41c4f4959b9b316cb112d0ada -> Peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Couldn't send request to peer a44f41b0b6644fb6aa29476aed8c964f. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:12.503840 31603 consensus_peers.cc:485] T eed5a5c41d9a46ce8a712b5b399e0472 P 05f4fac41c4f4959b9b316cb112d0ada -> Peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Couldn't send request to peer a44f41b0b6644fb6aa29476aed8c964f. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:12.508076 31706 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 21:27:12.543499 31860 consensus_peers.cc:485] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:27:12.546180 31863 consensus_peers.cc:485] T 2db9f537acba4424bca97ffc05c6d0b4 P 2d50f33557944bb480ea32199d9ee4f1 -> Peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Couldn't send request to peer a44f41b0b6644fb6aa29476aed8c964f. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:12.579793 1770 consensus_peers.cc:485] T acb3bdc518b84b5abc362cc4e12027c3 P 1e04157b98dd4fad98858c290b73c3db -> Peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Couldn't send request to peer a44f41b0b6644fb6aa29476aed8c964f. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:12.599468 2164 consensus_peers.cc:485] T 5ddffc460e0f42e4afc2de8e6b4d3433 P d1406e92aec84ac585304e57bd37a40d -> Peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Couldn't send request to peer a44f41b0b6644fb6aa29476aed8c964f. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:12.665339 31733 consensus_peers.cc:485] T 62e70b8e69c94d76b23c655a3f5de5bd P 08d7a0680d244b55b93d17f9ab57b84c -> Peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Couldn't send request to peer a44f41b0b6644fb6aa29476aed8c964f. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:12.692584 2164 consensus_peers.cc:485] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d -> Peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Couldn't send request to peer a44f41b0b6644fb6aa29476aed8c964f. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:12.696813 1767 consensus_peers.cc:485] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:27:12.708262 2164 consensus_peers.cc:485] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d -> Peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Couldn't send request to peer a44f41b0b6644fb6aa29476aed8c964f. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:12.719832 2030 consensus_peers.cc:485] T 4ed9c8356c4c4a5e94b755d58b3eb7a0 P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:27:12.839588 31600 consensus_peers.cc:485] T eed5a5c41d9a46ce8a712b5b399e0472 P 05f4fac41c4f4959b9b316cb112d0ada -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:27:12.847941 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 31585
W20240502 21:27:12.852156 31863 consensus_peers.cc:485] T e2ed57eb07384dc8953715c2c71b78b8 P 2d50f33557944bb480ea32199d9ee4f1 -> Peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Couldn't send request to peer a44f41b0b6644fb6aa29476aed8c964f. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:12.895924 2163 connection.cc:535] server connection from 127.24.226.130:56069 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:12.895975 31863 connection.cc:535] client connection to 127.24.226.130:37645 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:12.896168 31730 connection.cc:535] server connection from 127.24.226.130:55955 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:12.896284 2033 connection.cc:535] server connection from 127.24.226.130:60987 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:12.896417 1770 connection.cc:535] client connection to 127.24.226.130:37645 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:12.896560 31381 connection.cc:535] server connection from 127.24.226.130:49213 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:27:12.896750 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 31715
W20240502 21:27:12.897830 31863 consensus_peers.cc:485] T 7cc4bbc0deb8464db065e23cda176437 P 2d50f33557944bb480ea32199d9ee4f1 -> Peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): Couldn't send request to peer 05f4fac41c4f4959b9b316cb112d0ada. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.130:37645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:12.898049 1770 consensus_peers.cc:485] T 04c07a003bfe472e9142df6082759ae1 P 1e04157b98dd4fad98858c290b73c3db -> Peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): Couldn't send request to peer 05f4fac41c4f4959b9b316cb112d0ada. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.130:37645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:12.898170 1770 consensus_peers.cc:485] T 5d045ba2da9e482da25a39a0a2f38ff5 P 1e04157b98dd4fad98858c290b73c3db -> Peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): Couldn't send request to peer 05f4fac41c4f4959b9b316cb112d0ada. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.130:37645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:12.907263 31836 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 21:27:12.924265 2033 consensus_peers.cc:485] T 2c737c49a32b4f3ea5ef735fa254d48e P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): Couldn't send request to peer 05f4fac41c4f4959b9b316cb112d0ada. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.130:37645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:12.950924 2033 consensus_peers.cc:485] T 4117f23449114eaa814df89f80223e5e P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): Couldn't send request to peer 05f4fac41c4f4959b9b316cb112d0ada. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.130:37645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:12.952878 31860 consensus_peers.cc:485] T 41029f1332c64ba58ee0ee45714ba2a8 P 2d50f33557944bb480ea32199d9ee4f1 -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:27:12.971024 2164 consensus_peers.cc:485] T a8328088928f4de19f7cf20189382d91 P d1406e92aec84ac585304e57bd37a40d -> Peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Couldn't send request to peer a44f41b0b6644fb6aa29476aed8c964f. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:13.007452 1770 consensus_peers.cc:485] T 577e7a586ad9459f8e77ce4b8f563c09 P 1e04157b98dd4fad98858c290b73c3db -> Peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): Couldn't send request to peer 05f4fac41c4f4959b9b316cb112d0ada. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.130:37645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:13.052943 2161 consensus_peers.cc:485] T a8684b1ca3a642688a232afaf1b28f96 P d1406e92aec84ac585304e57bd37a40d -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:27:13.099282 1770 consensus_peers.cc:485] T b9342dbe0497499ba4c238f2ed033293 P 1e04157b98dd4fad98858c290b73c3db -> Peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): Couldn't send request to peer 05f4fac41c4f4959b9b316cb112d0ada. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.130:37645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:13.099931 31863 consensus_peers.cc:485] T f9bbc56ba2f841b9bb36a864e1a1f808 P 2d50f33557944bb480ea32199d9ee4f1 -> Peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): Couldn't send request to peer 05f4fac41c4f4959b9b316cb112d0ada. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.130:37645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:13.121695 2033 consensus_peers.cc:485] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): Couldn't send request to peer 05f4fac41c4f4959b9b316cb112d0ada. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.130:37645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:13.211400 6346 raft_consensus.cc:482] T 1ed225bdefc64ddd8eef2109bfd969df P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a44f41b0b6644fb6aa29476aed8c964f)
I20240502 21:27:13.211624 6346 raft_consensus.cc:504] T 1ed225bdefc64ddd8eef2109bfd969df P 08d7a0680d244b55b93d17f9ab57b84c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } }
I20240502 21:27:13.216361 6346 leader_election.cc:288] T 1ed225bdefc64ddd8eef2109bfd969df P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
W20240502 21:27:13.217175 31863 consensus_peers.cc:485] T 1bf575f91df54d75a9ac519924fb62e7 P 2d50f33557944bb480ea32199d9ee4f1 -> Peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): Couldn't send request to peer 05f4fac41c4f4959b9b316cb112d0ada. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.130:37645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:13.218513 2030 consensus_peers.cc:485] T f02f83ec0a494da5b66241d9b379a03f P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:27:13.223779 6348 raft_consensus.cc:482] T 1ed225bdefc64ddd8eef2109bfd969df P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a44f41b0b6644fb6aa29476aed8c964f)
I20240502 21:27:13.223968 6348 raft_consensus.cc:504] T 1ed225bdefc64ddd8eef2109bfd969df P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } }
I20240502 21:27:13.224689 6348 leader_election.cc:288] T 1ed225bdefc64ddd8eef2109bfd969df P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
W20240502 21:27:13.225936 31733 consensus_peers.cc:485] T b8db5de7df864ab983504e96fb738106 P 08d7a0680d244b55b93d17f9ab57b84c -> Peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): Couldn't send request to peer 05f4fac41c4f4959b9b316cb112d0ada. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.130:37645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:13.226059 31733 consensus_peers.cc:485] T 7b7b92c3d32844a99a9efe8a5bcc5a3d P 08d7a0680d244b55b93d17f9ab57b84c -> Peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): Couldn't send request to peer 05f4fac41c4f4959b9b316cb112d0ada. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.130:37645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:13.226138 31733 consensus_peers.cc:485] T 44277dc39b8a4729b704986edbfb9b84 P 08d7a0680d244b55b93d17f9ab57b84c -> Peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): Couldn't send request to peer 05f4fac41c4f4959b9b316cb112d0ada. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.130:37645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:13.226207 31733 consensus_peers.cc:485] T c5c344f8f84f4a73b853b4f908b93964 P 08d7a0680d244b55b93d17f9ab57b84c -> Peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): Couldn't send request to peer 05f4fac41c4f4959b9b316cb112d0ada. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.130:37645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:13.226282 31733 consensus_peers.cc:485] T fc93b5adc1d94e809e97e56b56822611 P 08d7a0680d244b55b93d17f9ab57b84c -> Peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): Couldn't send request to peer 05f4fac41c4f4959b9b316cb112d0ada. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.130:37645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:13.226471 31733 leader_election.cc:334] T 1ed225bdefc64ddd8eef2109bfd969df P 08d7a0680d244b55b93d17f9ab57b84c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111)
W20240502 21:27:13.230242 2164 leader_election.cc:334] T 1ed225bdefc64ddd8eef2109bfd969df P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111)
I20240502 21:27:13.230345 2224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ed225bdefc64ddd8eef2109bfd969df" candidate_uuid: "08d7a0680d244b55b93d17f9ab57b84c" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:27:13.230563 2224 raft_consensus.cc:2454] T 1ed225bdefc64ddd8eef2109bfd969df P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d7a0680d244b55b93d17f9ab57b84c in term 1.
I20240502 21:27:13.231850 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 31715
W20240502 21:27:13.246994 31863 consensus_peers.cc:485] T f1815f008fc041adb2da8c1e626a354f P 2d50f33557944bb480ea32199d9ee4f1 -> Peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): Couldn't send request to peer 05f4fac41c4f4959b9b316cb112d0ada. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.130:37645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:13.252375 6357 raft_consensus.cc:482] T 8ab82de23912483ea178376a5ba5cdc1 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a44f41b0b6644fb6aa29476aed8c964f)
I20240502 21:27:13.252568 6357 raft_consensus.cc:504] T 8ab82de23912483ea178376a5ba5cdc1 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } }
I20240502 21:27:13.253352 6357 leader_election.cc:288] T 8ab82de23912483ea178376a5ba5cdc1 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
W20240502 21:27:13.254519 2164 leader_election.cc:334] T 8ab82de23912483ea178376a5ba5cdc1 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111)
W20240502 21:27:13.258661 2033 consensus_peers.cc:485] T 9c40a6852649404f8928c2642e8ac3be P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): Couldn't send request to peer 05f4fac41c4f4959b9b316cb112d0ada. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.130:37645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:13.263026 6345 raft_consensus.cc:482] T df919f52065f444fbc2e9fb59a0494a5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a44f41b0b6644fb6aa29476aed8c964f)
I20240502 21:27:13.263217 6345 raft_consensus.cc:504] T df919f52065f444fbc2e9fb59a0494a5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } }
I20240502 21:27:13.263929 6345 leader_election.cc:288] T df919f52065f444fbc2e9fb59a0494a5 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:27:13.264505 31925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df919f52065f444fbc2e9fb59a0494a5" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2d50f33557944bb480ea32199d9ee4f1" is_pre_election: true
W20240502 21:27:13.265101 2033 leader_election.cc:334] T df919f52065f444fbc2e9fb59a0494a5 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111)
I20240502 21:27:13.265202 2033 leader_election.cc:302] T df919f52065f444fbc2e9fb59a0494a5 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b; no voters: 2d50f33557944bb480ea32199d9ee4f1, a44f41b0b6644fb6aa29476aed8c964f
I20240502 21:27:13.265444 6345 raft_consensus.cc:2726] T df919f52065f444fbc2e9fb59a0494a5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:27:13.273087 2164 consensus_peers.cc:485] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d -> Peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): Couldn't send request to peer 05f4fac41c4f4959b9b316cb112d0ada. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.130:37645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:13.275676 6270 raft_consensus.cc:482] T f8c1b589eead429089f20b013d4e5a0f P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a44f41b0b6644fb6aa29476aed8c964f)
I20240502 21:27:13.275816 6270 raft_consensus.cc:504] T f8c1b589eead429089f20b013d4e5a0f P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:13.276394 6270 leader_election.cc:288] T f8c1b589eead429089f20b013d4e5a0f P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:27:13.276862 1834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c1b589eead429089f20b013d4e5a0f" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
W20240502 21:27:13.278005 31863 leader_election.cc:334] T f8c1b589eead429089f20b013d4e5a0f P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111)
I20240502 21:27:13.278124 31863 leader_election.cc:302] T f8c1b589eead429089f20b013d4e5a0f P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d50f33557944bb480ea32199d9ee4f1; no voters: 1e04157b98dd4fad98858c290b73c3db, a44f41b0b6644fb6aa29476aed8c964f
I20240502 21:27:13.278367 6270 raft_consensus.cc:2726] T f8c1b589eead429089f20b013d4e5a0f P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:27:13.279551 6357 raft_consensus.cc:482] T 90af70ade50047d79664e63328101666 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a44f41b0b6644fb6aa29476aed8c964f)
I20240502 21:27:13.279724 6357 raft_consensus.cc:504] T 90af70ade50047d79664e63328101666 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:13.280246 6357 leader_election.cc:288] T 90af70ade50047d79664e63328101666 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135:38569)
I20240502 21:27:13.280786 2096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90af70ade50047d79664e63328101666" candidate_uuid: "d1406e92aec84ac585304e57bd37a40d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" is_pre_election: true
W20240502 21:27:13.281558 2164 leader_election.cc:334] T 90af70ade50047d79664e63328101666 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111)
I20240502 21:27:13.281715 2164 leader_election.cc:302] T 90af70ade50047d79664e63328101666 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1406e92aec84ac585304e57bd37a40d; no voters: 1ade0354c44c4c37ae480c7a94b11e1b, a44f41b0b6644fb6aa29476aed8c964f
I20240502 21:27:13.281988 6357 raft_consensus.cc:2726] T 90af70ade50047d79664e63328101666 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:27:13.282501 1769 connection.cc:535] client connection to 127.24.226.131:40807 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:13.282562 2163 connection.cc:535] client connection to 127.24.226.131:40807 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:13.282637 2030 connection.cc:535] server connection from 127.24.226.131:60059 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:13.282841 2032 connection.cc:535] client connection to 127.24.226.131:40807 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:13.282842 2161 connection.cc:535] server connection from 127.24.226.131:53089 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:13.283201 1768 connection.cc:535] server connection from 127.24.226.131:42585 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:27:13.283255 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 31845
W20240502 21:27:13.284157 1769 consensus_peers.cc:485] T e318adedad5a46ae9bca943b3cf9d818 P 1e04157b98dd4fad98858c290b73c3db -> Peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): Couldn't send request to peer 08d7a0680d244b55b93d17f9ab57b84c. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.131:40807: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:13.284168 2032 consensus_peers.cc:485] T c8b447d3c1a24051a914b7ed642808fa P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): Couldn't send request to peer 08d7a0680d244b55b93d17f9ab57b84c. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.131:40807: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:13.284324 2032 consensus_peers.cc:485] T f74ea10ab8a2459794f816707b87f494 P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): Couldn't send request to peer 08d7a0680d244b55b93d17f9ab57b84c. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.131:40807: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:13.284953 2163 consensus_peers.cc:485] T 3d024593597a4230b5c25b806843b779 P d1406e92aec84ac585304e57bd37a40d -> Peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): Couldn't send request to peer 08d7a0680d244b55b93d17f9ab57b84c. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.131:40807: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:13.285089 2163 consensus_peers.cc:485] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d -> Peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): Couldn't send request to peer 08d7a0680d244b55b93d17f9ab57b84c. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.131:40807: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:13.285190 2163 leader_election.cc:334] T 1ed225bdefc64ddd8eef2109bfd969df P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): Network error: Client connection negotiation failed: client connection to 127.24.226.131:40807: connect: Connection refused (error 111)
I20240502 21:27:13.285319 2163 leader_election.cc:302] T 1ed225bdefc64ddd8eef2109bfd969df P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1406e92aec84ac585304e57bd37a40d; no voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f
I20240502 21:27:13.285578 6357 raft_consensus.cc:2726] T 1ed225bdefc64ddd8eef2109bfd969df P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:27:13.285616 2163 leader_election.cc:334] T 8ab82de23912483ea178376a5ba5cdc1 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): Network error: Client connection negotiation failed: client connection to 127.24.226.131:40807: connect: Connection refused (error 111)
I20240502 21:27:13.285763 2163 leader_election.cc:302] T 8ab82de23912483ea178376a5ba5cdc1 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1406e92aec84ac585304e57bd37a40d; no voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f
I20240502 21:27:13.286000 6357 raft_consensus.cc:2726] T 8ab82de23912483ea178376a5ba5cdc1 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:27:13.286065 2163 consensus_peers.cc:485] T 0d923caa3f034a899050fe62dfc0c3d8 P d1406e92aec84ac585304e57bd37a40d -> Peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): Couldn't send request to peer 08d7a0680d244b55b93d17f9ab57b84c. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.131:40807: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:13.289582 31862 consensus_peers.cc:485] T 7cc4bbc0deb8464db065e23cda176437 P 2d50f33557944bb480ea32199d9ee4f1 -> Peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): Couldn't send request to peer 08d7a0680d244b55b93d17f9ab57b84c. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.131:40807: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:13.290520 6357 raft_consensus.cc:482] T b20df8896a2b423cb53c2c1a2a1b98e2 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a44f41b0b6644fb6aa29476aed8c964f)
I20240502 21:27:13.290647 6357 raft_consensus.cc:504] T b20df8896a2b423cb53c2c1a2a1b98e2 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } }
I20240502 21:27:13.291025 6357 leader_election.cc:288] T b20df8896a2b423cb53c2c1a2a1b98e2 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
W20240502 21:27:13.292142 2163 leader_election.cc:334] T b20df8896a2b423cb53c2c1a2a1b98e2 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): Network error: Client connection negotiation failed: client connection to 127.24.226.131:40807: connect: Connection refused (error 111)
W20240502 21:27:13.292479 2164 leader_election.cc:334] T b20df8896a2b423cb53c2c1a2a1b98e2 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111)
I20240502 21:27:13.292598 2164 leader_election.cc:302] T b20df8896a2b423cb53c2c1a2a1b98e2 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1406e92aec84ac585304e57bd37a40d; no voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f
I20240502 21:27:13.292894 6357 raft_consensus.cc:2726] T b20df8896a2b423cb53c2c1a2a1b98e2 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:27:13.294158 31967 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 21:27:13.346800 2163 consensus_peers.cc:485] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d -> Peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): Couldn't send request to peer 08d7a0680d244b55b93d17f9ab57b84c. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.131:40807: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:13.349767 1767 consensus_peers.cc:485] T c977ca41b72d4d45ac7146fbca350824 P 1e04157b98dd4fad98858c290b73c3db -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 21:27:13.375793 1769 consensus_peers.cc:485] T b67096e38f2e4c42a33e74f0145a04cc P 1e04157b98dd4fad98858c290b73c3db -> Peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): Couldn't send request to peer 08d7a0680d244b55b93d17f9ab57b84c. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.131:40807: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:13.380630 6355 raft_consensus.cc:482] T 988a0a9425434cd0b724710e9ebf23a6 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a44f41b0b6644fb6aa29476aed8c964f)
I20240502 21:27:13.380753 6355 raft_consensus.cc:504] T 988a0a9425434cd0b724710e9ebf23a6 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } }
I20240502 21:27:13.381331 6355 leader_election.cc:288] T 988a0a9425434cd0b724710e9ebf23a6 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:27:13.382220 6118 raft_consensus.cc:482] T 056db9c270e94ba58d356f7eaff5c124 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a44f41b0b6644fb6aa29476aed8c964f)
W20240502 21:27:13.382284 1770 leader_election.cc:334] T 988a0a9425434cd0b724710e9ebf23a6 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111)
I20240502 21:27:13.382370 6118 raft_consensus.cc:504] T 056db9c270e94ba58d356f7eaff5c124 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:13.382843 6118 leader_election.cc:288] T 056db9c270e94ba58d356f7eaff5c124 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
W20240502 21:27:13.384763 2033 leader_election.cc:334] T 056db9c270e94ba58d356f7eaff5c124 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111)
W20240502 21:27:13.386214 2032 leader_election.cc:334] T 056db9c270e94ba58d356f7eaff5c124 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): Network error: Client connection negotiation failed: client connection to 127.24.226.131:40807: connect: Connection refused (error 111)
I20240502 21:27:13.386353 2032 leader_election.cc:302] T 056db9c270e94ba58d356f7eaff5c124 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b; no voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f
I20240502 21:27:13.386608 6118 raft_consensus.cc:2726] T 056db9c270e94ba58d356f7eaff5c124 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:27:13.391019 6118 raft_consensus.cc:482] T 0d0c0636f5534b4389c5b09728c697e7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a44f41b0b6644fb6aa29476aed8c964f)
W20240502 21:27:13.391115 2032 consensus_peers.cc:485] T 2c737c49a32b4f3ea5ef735fa254d48e P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): Couldn't send request to peer 08d7a0680d244b55b93d17f9ab57b84c. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.131:40807: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:13.391191 6118 raft_consensus.cc:504] T 0d0c0636f5534b4389c5b09728c697e7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } }
I20240502 21:27:13.391634 6118 leader_election.cc:288] T 0d0c0636f5534b4389c5b09728c697e7 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
W20240502 21:27:13.392832 2032 leader_election.cc:334] T 0d0c0636f5534b4389c5b09728c697e7 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): Network error: Client connection negotiation failed: client connection to 127.24.226.131:40807: connect: Connection refused (error 111)
W20240502 21:27:13.392962 2033 leader_election.cc:334] T 0d0c0636f5534b4389c5b09728c697e7 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111)
I20240502 21:27:13.393097 2033 leader_election.cc:302] T 0d0c0636f5534b4389c5b09728c697e7 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b; no voters: 08d7a0680d244b55b93d17f9ab57b84c, a44f41b0b6644fb6aa29476aed8c964f
I20240502 21:27:13.393314 6118 raft_consensus.cc:2726] T 0d0c0636f5534b4389c5b09728c697e7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:27:13.437536 6118 raft_consensus.cc:482] T 90af70ade50047d79664e63328101666 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a44f41b0b6644fb6aa29476aed8c964f)
I20240502 21:27:13.437700 6118 raft_consensus.cc:504] T 90af70ade50047d79664e63328101666 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:13.438290 6118 leader_election.cc:288] T 90af70ade50047d79664e63328101666 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
I20240502 21:27:13.438681 6347 raft_consensus.cc:482] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a44f41b0b6644fb6aa29476aed8c964f)
I20240502 21:27:13.438846 6347 raft_consensus.cc:504] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } }
I20240502 21:27:13.439447 6347 leader_election.cc:288] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
I20240502 21:27:13.439658 2230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90af70ade50047d79664e63328101666" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d" is_pre_election: true
I20240502 21:27:13.439880 2230 raft_consensus.cc:2454] T 90af70ade50047d79664e63328101666 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 1.
W20240502 21:27:13.440297 2033 leader_election.cc:334] T 90af70ade50047d79664e63328101666 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111)
W20240502 21:27:13.440989 1770 leader_election.cc:334] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111)
I20240502 21:27:13.441293 2032 leader_election.cc:302] T 90af70ade50047d79664e63328101666 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, d1406e92aec84ac585304e57bd37a40d; no voters: a44f41b0b6644fb6aa29476aed8c964f
W20240502 21:27:13.441550 1767 leader_election.cc:334] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:13.441674 6118 raft_consensus.cc:2781] T 90af70ade50047d79664e63328101666 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:27:13.441675 1767 leader_election.cc:302] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db; no voters: 9dec645b473f40578e726f3e50e219bf, a44f41b0b6644fb6aa29476aed8c964f
I20240502 21:27:13.441754 6118 raft_consensus.cc:482] T 90af70ade50047d79664e63328101666 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting leader election (detected failure of leader a44f41b0b6644fb6aa29476aed8c964f)
I20240502 21:27:13.441846 6118 raft_consensus.cc:3037] T 90af70ade50047d79664e63328101666 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:13.441885 6347 raft_consensus.cc:2726] T f13c3b9c4b7c471fa34156b781473ec4 P 1e04157b98dd4fad98858c290b73c3db [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 21:27:13.442824 2030 consensus_peers.cc:485] T 565f30d5539c4b7ea956dc49d0f9a55e P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 21:27:13.443960 6118 raft_consensus.cc:504] T 90af70ade50047d79664e63328101666 P 1ade0354c44c4c37ae480c7a94b11e1b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:13.444380 6118 leader_election.cc:288] T 90af70ade50047d79664e63328101666 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 election: Requested vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), d1406e92aec84ac585304e57bd37a40d (127.24.226.136:34907)
W20240502 21:27:13.445564 2033 leader_election.cc:334] T 90af70ade50047d79664e63328101666 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111)
I20240502 21:27:13.446867 2230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90af70ade50047d79664e63328101666" candidate_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1406e92aec84ac585304e57bd37a40d"
I20240502 21:27:13.447067 2230 raft_consensus.cc:3037] T 90af70ade50047d79664e63328101666 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:13.448851 2230 raft_consensus.cc:2454] T 90af70ade50047d79664e63328101666 P d1406e92aec84ac585304e57bd37a40d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ade0354c44c4c37ae480c7a94b11e1b in term 2.
I20240502 21:27:13.449263 2032 leader_election.cc:302] T 90af70ade50047d79664e63328101666 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b, d1406e92aec84ac585304e57bd37a40d; no voters: a44f41b0b6644fb6aa29476aed8c964f
I20240502 21:27:13.449573 6118 raft_consensus.cc:2781] T 90af70ade50047d79664e63328101666 P 1ade0354c44c4c37ae480c7a94b11e1b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:13.449719 6118 raft_consensus.cc:686] T 90af70ade50047d79664e63328101666 P 1ade0354c44c4c37ae480c7a94b11e1b [term 2 LEADER]: Becoming Leader. State: Replica: 1ade0354c44c4c37ae480c7a94b11e1b, State: Running, Role: LEADER
I20240502 21:27:13.449949 6118 consensus_queue.cc:237] T 90af70ade50047d79664e63328101666 P 1ade0354c44c4c37ae480c7a94b11e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } }
I20240502 21:27:13.454031 31389 catalog_manager.cc:5484] T 90af70ade50047d79664e63328101666 P 1ade0354c44c4c37ae480c7a94b11e1b reported cstate change: term changed from 1 to 2, leader changed from a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129) to 1ade0354c44c4c37ae480c7a94b11e1b (127.24.226.135). New cstate: current_term: 2 leader_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:13.520378 5945 raft_consensus.cc:482] T 3e6657723fe84e749dea768de44638e7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a44f41b0b6644fb6aa29476aed8c964f)
I20240502 21:27:13.520579 5945 raft_consensus.cc:504] T 3e6657723fe84e749dea768de44638e7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:27:13.521173 5945 leader_election.cc:288] T 3e6657723fe84e749dea768de44638e7 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
W20240502 21:27:13.522193 2033 leader_election.cc:334] T 3e6657723fe84e749dea768de44638e7 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111)
W20240502 21:27:13.522375 2030 leader_election.cc:334] T 3e6657723fe84e749dea768de44638e7 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:13.522487 2030 leader_election.cc:302] T 3e6657723fe84e749dea768de44638e7 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b; no voters: 9dec645b473f40578e726f3e50e219bf, a44f41b0b6644fb6aa29476aed8c964f
I20240502 21:27:13.522680 5945 raft_consensus.cc:2726] T 3e6657723fe84e749dea768de44638e7 P 1ade0354c44c4c37ae480c7a94b11e1b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 21:27:13.554558 6355 raft_consensus.cc:482] T 381ac1d714204cef86c8e1a73e24b863 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a44f41b0b6644fb6aa29476aed8c964f)
I20240502 21:27:13.554736 6355 raft_consensus.cc:504] T 381ac1d714204cef86c8e1a73e24b863 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } }
I20240502 21:27:13.555280 6355 leader_election.cc:288] T 381ac1d714204cef86c8e1a73e24b863 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
W20240502 21:27:13.556789 1770 leader_election.cc:334] T 381ac1d714204cef86c8e1a73e24b863 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111)
W20240502 21:27:13.560197 2161 consensus_peers.cc:485] T 2068521c26dc44f9ba60e274eb039b22 P d1406e92aec84ac585304e57bd37a40d -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 21:27:13.566236 6348 raft_consensus.cc:482] T 55fc18e4cc98446ca50c1dd2b6a9a89d P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a44f41b0b6644fb6aa29476aed8c964f)
I20240502 21:27:13.566418 6348 raft_consensus.cc:504] T 55fc18e4cc98446ca50c1dd2b6a9a89d P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } attrs { promote: false } }
I20240502 21:27:13.567106 6348 leader_election.cc:288] T 55fc18e4cc98446ca50c1dd2b6a9a89d P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
W20240502 21:27:13.568197 2164 leader_election.cc:334] T 55fc18e4cc98446ca50c1dd2b6a9a89d P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111)
W20240502 21:27:13.568334 2161 leader_election.cc:334] T 55fc18e4cc98446ca50c1dd2b6a9a89d P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:13.568497 2161 leader_election.cc:302] T 55fc18e4cc98446ca50c1dd2b6a9a89d P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1406e92aec84ac585304e57bd37a40d; no voters: 9dec645b473f40578e726f3e50e219bf, a44f41b0b6644fb6aa29476aed8c964f
I20240502 21:27:13.568737 6348 raft_consensus.cc:2726] T 55fc18e4cc98446ca50c1dd2b6a9a89d P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:27:13.582056 2161 consensus_peers.cc:485] T 1022c4c6ee8642ca95250933a940fe7a P d1406e92aec84ac585304e57bd37a40d -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 21:27:13.589311 6270 raft_consensus.cc:482] T 759b7529d7d84baab1a4622d34f803bf P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a44f41b0b6644fb6aa29476aed8c964f)
I20240502 21:27:13.589468 6270 raft_consensus.cc:504] T 759b7529d7d84baab1a4622d34f803bf P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } attrs { promote: false } }
I20240502 21:27:13.589725 6358 raft_consensus.cc:482] T 381ac1d714204cef86c8e1a73e24b863 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a44f41b0b6644fb6aa29476aed8c964f)
I20240502 21:27:13.589828 6358 raft_consensus.cc:504] T 381ac1d714204cef86c8e1a73e24b863 P 2d50f33557944bb480ea32199d9ee4f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } attrs { promote: false } }
I20240502 21:27:13.590108 6270 leader_election.cc:288] T 759b7529d7d84baab1a4622d34f803bf P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753)
I20240502 21:27:13.590296 6358 leader_election.cc:288] T 381ac1d714204cef86c8e1a73e24b863 P 2d50f33557944bb480ea32199d9ee4f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863)
I20240502 21:27:13.594190 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 31845
I20240502 21:27:13.596729 6348 raft_consensus.cc:482] T 51551a1967b74094b373cc030e2bf850 P d1406e92aec84ac585304e57bd37a40d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a44f41b0b6644fb6aa29476aed8c964f)
I20240502 21:27:13.596891 6348 raft_consensus.cc:504] T 51551a1967b74094b373cc030e2bf850 P d1406e92aec84ac585304e57bd37a40d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } attrs { promote: false } }
I20240502 21:27:13.597482 6348 leader_election.cc:288] T 51551a1967b74094b373cc030e2bf850 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
W20240502 21:27:13.598757 2164 leader_election.cc:334] T 51551a1967b74094b373cc030e2bf850 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111)
W20240502 21:27:13.600392 2161 leader_election.cc:334] T 51551a1967b74094b373cc030e2bf850 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:13.600535 2161 leader_election.cc:302] T 51551a1967b74094b373cc030e2bf850 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1406e92aec84ac585304e57bd37a40d; no voters: 9dec645b473f40578e726f3e50e219bf, a44f41b0b6644fb6aa29476aed8c964f
I20240502 21:27:13.600744 6357 raft_consensus.cc:2726] T 51551a1967b74094b373cc030e2bf850 P d1406e92aec84ac585304e57bd37a40d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 21:27:13.608597 1831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "381ac1d714204cef86c8e1a73e24b863" candidate_uuid: "2d50f33557944bb480ea32199d9ee4f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "1e04157b98dd4fad98858c290b73c3db" is_pre_election: true
I20240502 21:27:13.608851 1831 raft_consensus.cc:2454] T 381ac1d714204cef86c8e1a73e24b863 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d50f33557944bb480ea32199d9ee4f1 in term 1.
I20240502 21:27:13.609288 6118 raft_consensus.cc:482] T 4bc968ae4dcc4636a49257b6d5824bee P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a44f41b0b6644fb6aa29476aed8c964f)
I20240502 21:27:13.609457 6118 raft_consensus.cc:504] T 4bc968ae4dcc4636a49257b6d5824bee P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } attrs { promote: false } }
I20240502 21:27:13.610181 6118 leader_election.cc:288] T 4bc968ae4dcc4636a49257b6d5824bee P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
I20240502 21:27:13.610374 6355 raft_consensus.cc:482] T f8c1b589eead429089f20b013d4e5a0f P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a44f41b0b6644fb6aa29476aed8c964f)
I20240502 21:27:13.610505 6355 raft_consensus.cc:504] T f8c1b589eead429089f20b013d4e5a0f P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "a44f41b0b6644fb6aa29476aed8c964f" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 41039 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } }
I20240502 21:27:13.611116 6355 leader_election.cc:288] T f8c1b589eead429089f20b013d4e5a0f P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039)
W20240502 21:27:13.611516 2033 leader_election.cc:334] T 4bc968ae4dcc4636a49257b6d5824bee P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111)
W20240502 21:27:13.611696 2033 leader_election.cc:334] T 4bc968ae4dcc4636a49257b6d5824bee P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): Network error: Client connection negotiation failed: client connection to 127.24.226.130:37645: connect: Connection refused (error 111)
I20240502 21:27:13.611817 2033 leader_election.cc:302] T 4bc968ae4dcc4636a49257b6d5824bee P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b; no voters: 05f4fac41c4f4959b9b316cb112d0ada, a44f41b0b6644fb6aa29476aed8c964f
I20240502 21:27:13.612056 6118 raft_consensus.cc:2726] T 4bc968ae4dcc4636a49257b6d5824bee P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:27:13.612969 1770 leader_election.cc:334] T f8c1b589eead429089f20b013d4e5a0f P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111)
W20240502 21:27:13.642328 2163 connection.cc:535] server connection from 127.24.226.132:59119 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:13.642537 1770 connection.cc:535] client connection to 127.24.226.132:35933 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:13.642741 2164 connection.cc:535] client connection to 127.24.226.132:35933 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:27:13.642834 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 1752
W20240502 21:27:13.643640 1770 connection.cc:535] server connection from 127.24.226.132:54135 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:13.644326 2164 consensus_peers.cc:485] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d -> Peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): Couldn't send request to peer 2d50f33557944bb480ea32199d9ee4f1. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.132:35933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:13.644482 2164 consensus_peers.cc:485] T 3d024593597a4230b5c25b806843b779 P d1406e92aec84ac585304e57bd37a40d -> Peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): Couldn't send request to peer 2d50f33557944bb480ea32199d9ee4f1. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.132:35933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:13.644779 2030 connection.cc:535] server connection from 127.24.226.132:51939 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:13.644928 2033 connection.cc:535] client connection to 127.24.226.132:35933 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:13.645143 2164 consensus_peers.cc:485] T 77ee047b191e4400ab02040e3fbde8f6 P d1406e92aec84ac585304e57bd37a40d -> Peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): Couldn't send request to peer 2d50f33557944bb480ea32199d9ee4f1. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.132:35933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:13.645475 1770 leader_election.cc:334] T 381ac1d714204cef86c8e1a73e24b863 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): Network error: Client connection negotiation failed: client connection to 127.24.226.132:35933: connect: Connection refused (error 111)
I20240502 21:27:13.645622 1770 leader_election.cc:302] T 381ac1d714204cef86c8e1a73e24b863 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db; no voters: 2d50f33557944bb480ea32199d9ee4f1, a44f41b0b6644fb6aa29476aed8c964f
I20240502 21:27:13.645961 6355 raft_consensus.cc:2726] T 381ac1d714204cef86c8e1a73e24b863 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:27:13.646064 1770 leader_election.cc:334] T 988a0a9425434cd0b724710e9ebf23a6 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): Network error: Client connection negotiation failed: client connection to 127.24.226.132:35933: connect: Connection refused (error 111)
I20240502 21:27:13.646203 1770 leader_election.cc:302] T 988a0a9425434cd0b724710e9ebf23a6 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db; no voters: 2d50f33557944bb480ea32199d9ee4f1, a44f41b0b6644fb6aa29476aed8c964f
I20240502 21:27:13.646505 6355 raft_consensus.cc:2726] T 988a0a9425434cd0b724710e9ebf23a6 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:27:13.646603 1770 consensus_peers.cc:485] T 2e4ed0121a404869b6fa33b14d3c49f8 P 1e04157b98dd4fad98858c290b73c3db -> Peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): Couldn't send request to peer 2d50f33557944bb480ea32199d9ee4f1. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.132:35933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:13.646768 1770 leader_election.cc:334] T f8c1b589eead429089f20b013d4e5a0f P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): Network error: Client connection negotiation failed: client connection to 127.24.226.132:35933: connect: Connection refused (error 111)
I20240502 21:27:13.646893 1770 leader_election.cc:302] T f8c1b589eead429089f20b013d4e5a0f P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db; no voters: 2d50f33557944bb480ea32199d9ee4f1, a44f41b0b6644fb6aa29476aed8c964f
I20240502 21:27:13.647128 6355 raft_consensus.cc:2726] T f8c1b589eead429089f20b013d4e5a0f P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:27:13.647192 1770 consensus_peers.cc:485] T 4f09394db52e489e9bc42a4883ee8b1c P 1e04157b98dd4fad98858c290b73c3db -> Peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): Couldn't send request to peer 2d50f33557944bb480ea32199d9ee4f1. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.132:35933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:13.645251 2164 consensus_peers.cc:485] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d -> Peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): Couldn't send request to peer 2d50f33557944bb480ea32199d9ee4f1. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.132:35933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:13.648954 2033 consensus_peers.cc:485] T 6c84ffd75b4048d886210d6ecbe4d717 P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): Couldn't send request to peer 2d50f33557944bb480ea32199d9ee4f1. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.132:35933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:13.653945 1873 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 21:27:13.705104 2033 consensus_peers.cc:485] T c8b447d3c1a24051a914b7ed642808fa P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): Couldn't send request to peer 2d50f33557944bb480ea32199d9ee4f1. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.132:35933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:13.721313 2164 consensus_peers.cc:485] T c5ba5610b754466fb1e12db451295809 P d1406e92aec84ac585304e57bd37a40d -> Peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): Couldn't send request to peer 2d50f33557944bb480ea32199d9ee4f1. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.132:35933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:13.749444 31379 catalog_manager.cc:4573] TS 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 97a450582be947a48b4aa789389e9e5a: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
W20240502 21:27:13.772615 2164 consensus_peers.cc:485] T a699f448e116428f85cfee6ccbc98612 P d1406e92aec84ac585304e57bd37a40d -> Peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): Couldn't send request to peer 2d50f33557944bb480ea32199d9ee4f1. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.132:35933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:13.801465 6345 raft_consensus.cc:482] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 05f4fac41c4f4959b9b316cb112d0ada)
I20240502 21:27:13.801584 6345 raft_consensus.cc:504] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:13.802124 6345 leader_election.cc:288] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
W20240502 21:27:13.803157 2033 leader_election.cc:334] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): Network error: Client connection negotiation failed: client connection to 127.24.226.130:37645: connect: Connection refused (error 111)
W20240502 21:27:13.864919 2033 consensus_peers.cc:485] T 90af70ade50047d79664e63328101666 P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer a44f41b0b6644fb6aa29476aed8c964f (127.24.226.129:41039): Couldn't send request to peer a44f41b0b6644fb6aa29476aed8c964f. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.129:41039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:13.885334 2161 consensus_peers.cc:485] T 59f36d947f4341f39b6969cc1e065b6c P d1406e92aec84ac585304e57bd37a40d -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 21:27:13.897341 6345 raft_consensus.cc:482] T de2216a74f8344779a881651b66a85e5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 05f4fac41c4f4959b9b316cb112d0ada)
I20240502 21:27:13.897512 6345 raft_consensus.cc:504] T de2216a74f8344779a881651b66a85e5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "1ade0354c44c4c37ae480c7a94b11e1b" member_type: VOTER last_known_addr { host: "127.24.226.135" port: 38569 } } peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } attrs { promote: false } }
I20240502 21:27:13.898114 6345 leader_election.cc:288] T de2216a74f8344779a881651b66a85e5 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807)
W20240502 21:27:13.899114 2033 leader_election.cc:334] T de2216a74f8344779a881651b66a85e5 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): Network error: Client connection negotiation failed: client connection to 127.24.226.130:37645: connect: Connection refused (error 111)
W20240502 21:27:13.899333 2032 leader_election.cc:334] T de2216a74f8344779a881651b66a85e5 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807): Network error: Client connection negotiation failed: client connection to 127.24.226.131:40807: connect: Connection refused (error 111)
I20240502 21:27:13.899441 2032 leader_election.cc:302] T de2216a74f8344779a881651b66a85e5 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b; no voters: 05f4fac41c4f4959b9b316cb112d0ada, 08d7a0680d244b55b93d17f9ab57b84c
I20240502 21:27:13.899695 6345 raft_consensus.cc:2726] T de2216a74f8344779a881651b66a85e5 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:27:13.921583 6355 raft_consensus.cc:482] T f59b3b104a4647d59e0ffcfd6288d106 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 05f4fac41c4f4959b9b316cb112d0ada)
I20240502 21:27:13.921748 6355 raft_consensus.cc:504] T f59b3b104a4647d59e0ffcfd6288d106 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04157b98dd4fad98858c290b73c3db" member_type: VOTER last_known_addr { host: "127.24.226.133" port: 39863 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:13.922713 6355 leader_election.cc:288] T f59b3b104a4647d59e0ffcfd6288d106 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
W20240502 21:27:13.923660 1770 consensus_peers.cc:485] T c977ca41b72d4d45ac7146fbca350824 P 1e04157b98dd4fad98858c290b73c3db -> Peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): Couldn't send request to peer 2d50f33557944bb480ea32199d9ee4f1. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.132:35933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:13.924474 1770 leader_election.cc:334] T f59b3b104a4647d59e0ffcfd6288d106 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): Network error: Client connection negotiation failed: client connection to 127.24.226.132:35933: connect: Connection refused (error 111)
W20240502 21:27:13.924687 1770 leader_election.cc:334] T f59b3b104a4647d59e0ffcfd6288d106 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): Network error: Client connection negotiation failed: client connection to 127.24.226.130:37645: connect: Connection refused (error 111)
I20240502 21:27:13.924759 1770 leader_election.cc:302] T f59b3b104a4647d59e0ffcfd6288d106 P 1e04157b98dd4fad98858c290b73c3db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e04157b98dd4fad98858c290b73c3db; no voters: 05f4fac41c4f4959b9b316cb112d0ada, 2d50f33557944bb480ea32199d9ee4f1
I20240502 21:27:13.924963 6384 raft_consensus.cc:2726] T f59b3b104a4647d59e0ffcfd6288d106 P 1e04157b98dd4fad98858c290b73c3db [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:27:13.929675 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 1752
W20240502 21:27:13.967984 2161 connection.cc:535] server connection from 127.24.226.133:33731 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:13.968462 2031 connection.cc:535] server connection from 127.24.226.133:57191 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:27:13.968570 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 2014
W20240502 21:27:13.968616 2032 connection.cc:535] client connection to 127.24.226.133:39863 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:13.968685 2163 connection.cc:535] client connection to 127.24.226.133:39863 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:13.969674 2163 consensus_peers.cc:485] T 883c729aa26f47178f40a17ad3b7e590 P d1406e92aec84ac585304e57bd37a40d -> Peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): Couldn't send request to peer 1e04157b98dd4fad98858c290b73c3db. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.133:39863: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:13.970129 2032 leader_election.cc:334] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): Network error: Client connection negotiation failed: client connection to 127.24.226.133:39863: connect: Connection refused (error 111)
I20240502 21:27:13.970252 2032 leader_election.cc:302] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 1ade0354c44c4c37ae480c7a94b11e1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ade0354c44c4c37ae480c7a94b11e1b; no voters: 05f4fac41c4f4959b9b316cb112d0ada, 1e04157b98dd4fad98858c290b73c3db
I20240502 21:27:13.970468 6357 raft_consensus.cc:482] T 925797977edf495ca09aa25988d14fac P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 05f4fac41c4f4959b9b316cb112d0ada)
W20240502 21:27:13.970602 2032 consensus_peers.cc:485] T e9140579721c4b3fbba776e4e16e5d9d P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): Couldn't send request to peer 1e04157b98dd4fad98858c290b73c3db. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.133:39863: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:13.970603 6357 raft_consensus.cc:504] T 925797977edf495ca09aa25988d14fac P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dec645b473f40578e726f3e50e219bf" member_type: VOTER last_known_addr { host: "127.24.226.134" port: 43753 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } }
I20240502 21:27:13.970737 6345 raft_consensus.cc:2726] T d5f7ee5a6c8a41fc80a5ef1c65a75579 P 1ade0354c44c4c37ae480c7a94b11e1b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:27:13.971259 6357 leader_election.cc:288] T 925797977edf495ca09aa25988d14fac P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753), 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645)
W20240502 21:27:13.972633 2164 leader_election.cc:334] T 925797977edf495ca09aa25988d14fac P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): Network error: Client connection negotiation failed: client connection to 127.24.226.130:37645: connect: Connection refused (error 111)
W20240502 21:27:13.973012 2032 consensus_peers.cc:485] T 579fc086d7524e488f8dc4526b89597b P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): Couldn't send request to peer 1e04157b98dd4fad98858c290b73c3db. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.133:39863: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:13.973069 2161 leader_election.cc:334] T 925797977edf495ca09aa25988d14fac P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111)
I20240502 21:27:13.973183 2161 leader_election.cc:302] T 925797977edf495ca09aa25988d14fac P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1406e92aec84ac585304e57bd37a40d; no voters: 05f4fac41c4f4959b9b316cb112d0ada, 9dec645b473f40578e726f3e50e219bf
I20240502 21:27:13.973412 6357 raft_consensus.cc:2726] T 925797977edf495ca09aa25988d14fac P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:27:13.975662 2137 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 21:27:14.102420 2163 consensus_peers.cc:485] T 04070a3231524eac83e0c94bf2aee585 P d1406e92aec84ac585304e57bd37a40d -> Peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): Couldn't send request to peer 1e04157b98dd4fad98858c290b73c3db. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.133:39863: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:14.129587 2163 consensus_peers.cc:485] T f5e82c0ab5c441efbcfbcd383b8db82f P d1406e92aec84ac585304e57bd37a40d -> Peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): Couldn't send request to peer 1e04157b98dd4fad98858c290b73c3db. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.133:39863: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:14.163139 2163 consensus_peers.cc:485] T 84ba90ce8dd24ea2b325b1dba2e4c494 P d1406e92aec84ac585304e57bd37a40d -> Peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): Couldn't send request to peer 1e04157b98dd4fad98858c290b73c3db. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.133:39863: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:14.186040 2161 consensus_peers.cc:485] T a8328088928f4de19f7cf20189382d91 P d1406e92aec84ac585304e57bd37a40d -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:27:14.193361 2163 consensus_peers.cc:485] T 77ee047b191e4400ab02040e3fbde8f6 P d1406e92aec84ac585304e57bd37a40d -> Peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): Couldn't send request to peer 1e04157b98dd4fad98858c290b73c3db. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.133:39863: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:14.197428 2163 consensus_peers.cc:485] T a8684b1ca3a642688a232afaf1b28f96 P d1406e92aec84ac585304e57bd37a40d -> Peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): Couldn't send request to peer 1e04157b98dd4fad98858c290b73c3db. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.133:39863: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:14.219053 2163 consensus_peers.cc:485] T 5ddffc460e0f42e4afc2de8e6b4d3433 P d1406e92aec84ac585304e57bd37a40d -> Peer 1e04157b98dd4fad98858c290b73c3db (127.24.226.133:39863): Couldn't send request to peer 1e04157b98dd4fad98858c290b73c3db. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.133:39863: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:14.226043 2030 consensus_peers.cc:485] T 46279270b2334633906af57c4b43e969 P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 21:27:14.226248 2030 consensus_peers.cc:485] T b1d59ebecac34cf7b33424cc73cff030 P 1ade0354c44c4c37ae480c7a94b11e1b -> Peer 9dec645b473f40578e726f3e50e219bf (127.24.226.134:43753): Couldn't send request to peer 9dec645b473f40578e726f3e50e219bf. Status: Network error: Client connection negotiation failed: client connection to 127.24.226.134:43753: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 21:27:14.226920 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 2014
I20240502 21:27:14.228488 2224 raft_consensus.cc:1261] T 90af70ade50047d79664e63328101666 P d1406e92aec84ac585304e57bd37a40d [term 2 FOLLOWER]: Refusing update from remote peer 1ade0354c44c4c37ae480c7a94b11e1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240502 21:27:14.264314 2164 connection.cc:535] server connection from 127.24.226.135:40523 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:27:14.264595 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 2146
I20240502 21:27:14.270140 2270 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:27:14.511292 6357 raft_consensus.cc:482] T 49acc5301b9849e9b6795179acbc59a8 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 05f4fac41c4f4959b9b316cb112d0ada)
I20240502 21:27:14.511524 6357 raft_consensus.cc:504] T 49acc5301b9849e9b6795179acbc59a8 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "05f4fac41c4f4959b9b316cb112d0ada" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 37645 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } attrs { promote: false } }
I20240502 21:27:14.512616 6348 raft_consensus.cc:482] T 8702f1a38b274f08affe558f6ef62d74 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 08d7a0680d244b55b93d17f9ab57b84c)
I20240502 21:27:14.512777 6348 raft_consensus.cc:504] T 8702f1a38b274f08affe558f6ef62d74 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d7a0680d244b55b93d17f9ab57b84c" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 40807 } } peers { permanent_uuid: "d1406e92aec84ac585304e57bd37a40d" member_type: VOTER last_known_addr { host: "127.24.226.136" port: 34907 } } peers { permanent_uuid: "2d50f33557944bb480ea32199d9ee4f1" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 35933 } }
I20240502 21:27:14.512390 6357 leader_election.cc:288] T 49acc5301b9849e9b6795179acbc59a8 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
I20240502 21:27:14.513551 6348 leader_election.cc:288] T 8702f1a38b274f08affe558f6ef62d74 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08d7a0680d244b55b93d17f9ab57b84c (127.24.226.131:40807), 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933)
W20240502 21:27:14.517299 2164 leader_election.cc:334] T 49acc5301b9849e9b6795179acbc59a8 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): Network error: Client connection negotiation failed: client connection to 127.24.226.132:35933: connect: Connection refused (error 111)
W20240502 21:27:14.517537 2164 leader_election.cc:334] T 8702f1a38b274f08affe558f6ef62d74 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d50f33557944bb480ea32199d9ee4f1 (127.24.226.132:35933): Network error: Client connection negotiation failed: client connection to 127.24.226.132:35933: connect: Connection refused (error 111)
W20240502 21:27:14.518111 2164 leader_election.cc:334] T 49acc5301b9849e9b6795179acbc59a8 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 05f4fac41c4f4959b9b316cb112d0ada (127.24.226.130:37645): Network error: Client connection negotiation failed: client connection to 127.24.226.130:37645: connect: Connection refused (error 111)
I20240502 21:27:14.518224 2164 leader_election.cc:302] T 49acc5301b9849e9b6795179acbc59a8 P d1406e92aec84ac585304e57bd37a40d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1406e92aec84ac585304e57bd37a40d; no voters: 05f4fac41c4f4959b9b316cb112d0ada, 2d50f33557944bb480ea32199d9ee4f1
I20240502 21:27:14.518436 6348 raft_consensus.cc:2726] T 49acc5301b9849e9b6795179acbc59a8 P d1406e92aec84ac585304e57bd37a40d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:27:14.519646 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 2146
I20240502 21:27:14.557230 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 31364
I20240502 21:27:14.562567 31424 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:27:14.833935 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 31364
2024-05-02T21:27:14Z chronyd exiting
I20240502 21:27:14.891299 25482 test_util.cc:175] -----------------------------------------------
I20240502 21:27:14.891381 25482 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714685121724517-25482-0
[ FAILED ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/1, where GetParam() = 4-byte object <01-00 00-00> (72328 ms)
[----------] 1 test from DDLDuringRebalancingTest (72328 ms total)
[----------] 1 test from RebalancingDuringElectionStormTest
[ RUN ] RebalancingDuringElectionStormTest.RoundRobin/1
I20240502 21:27:14.893273 25482 test_util.cc:268] Using random seed: 1146343864
I20240502 21:27:14.894889 25482 ts_itest-base.cc:112] Starting cluster with:
I20240502 21:27:14.894955 25482 ts_itest-base.cc:113] --------------
I20240502 21:27:14.895000 25482 ts_itest-base.cc:114] 8 tablet servers
I20240502 21:27:14.895059 25482 ts_itest-base.cc:115] 3 replicas per TS
I20240502 21:27:14.895104 25482 ts_itest-base.cc:116] --------------
2024-05-02T21:27:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T21:27:14Z Disabled control of system clock
I20240502 21:27:14.922897 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.226.190:40767
--webserver_interface=127.24.226.190
--webserver_port=0
--builtin_ntp_servers=127.24.226.148:46251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.226.190:40767
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 21:27:15.224247 6408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:15.224524 6408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:27:15.224637 6408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:15.231117 6408 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:27:15.231233 6408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 21:27:15.231305 6408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:27:15.231376 6408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 21:27:15.231432 6408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 21:27:15.239133 6408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:46251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.24.226.190:40767
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.226.190:40767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.24.226.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:27:15.240125 6408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:27:15.241581 6408 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:15.250502 6413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:15.250597 6416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:15.250555 6414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:15.251430 6408 server_base.cc:1034] running on GCE node
I20240502 21:27:15.252009 6408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:27:15.253175 6408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:27:15.254371 6408 hybrid_clock.cc:648] HybridClock initialized: now 1714685235254326 us; error 33 us; skew 500 ppm
I20240502 21:27:15.254662 6408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:15.256868 6408 webserver.cc:460] Webserver started at http://127.24.226.190:33797/ using document root <none> and password file <none>
I20240502 21:27:15.257503 6408 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:15.257611 6408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:15.257885 6408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:27:15.262671 6408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "16ed06c661f4400891d60a84f376a981"
format_stamp: "Formatted at 2024-05-02 21:27:15 on dist-test-slave-41k1"
I20240502 21:27:15.263440 6408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "16ed06c661f4400891d60a84f376a981"
format_stamp: "Formatted at 2024-05-02 21:27:15 on dist-test-slave-41k1"
I20240502 21:27:15.268273 6408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20240502 21:27:15.271310 6422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:15.272408 6408 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:15.272603 6408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/wal
uuid: "16ed06c661f4400891d60a84f376a981"
format_stamp: "Formatted at 2024-05-02 21:27:15 on dist-test-slave-41k1"
I20240502 21:27:15.272764 6408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:15.295002 6408 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:15.295782 6408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:27:15.296066 6408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:15.313038 6408 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.190:40767
I20240502 21:27:15.313066 6473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.190:40767 every 8 connection(s)
I20240502 21:27:15.314280 6408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 21:27:15.317363 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 6408
I20240502 21:27:15.317564 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 21:27:15.317672 6474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:15.329192 6474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16ed06c661f4400891d60a84f376a981: Bootstrap starting.
I20240502 21:27:15.331885 6474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16ed06c661f4400891d60a84f376a981: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:15.332813 6474 log.cc:824] T 00000000000000000000000000000000 P 16ed06c661f4400891d60a84f376a981: Log is configured to *not* fsync() on all Append() calls
I20240502 21:27:15.334864 6474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16ed06c661f4400891d60a84f376a981: No bootstrap required, opened a new log
I20240502 21:27:15.340395 6474 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16ed06c661f4400891d60a84f376a981 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ed06c661f4400891d60a84f376a981" member_type: VOTER last_known_addr { host: "127.24.226.190" port: 40767 } }
I20240502 21:27:15.340729 6474 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16ed06c661f4400891d60a84f376a981 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:15.340827 6474 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16ed06c661f4400891d60a84f376a981 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16ed06c661f4400891d60a84f376a981, State: Initialized, Role: FOLLOWER
I20240502 21:27:15.341324 6474 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16ed06c661f4400891d60a84f376a981 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ed06c661f4400891d60a84f376a981" member_type: VOTER last_known_addr { host: "127.24.226.190" port: 40767 } }
I20240502 21:27:15.341531 6474 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16ed06c661f4400891d60a84f376a981 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:27:15.341651 6474 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16ed06c661f4400891d60a84f376a981 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:27:15.341782 6474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16ed06c661f4400891d60a84f376a981 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:15.343488 6474 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16ed06c661f4400891d60a84f376a981 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ed06c661f4400891d60a84f376a981" member_type: VOTER last_known_addr { host: "127.24.226.190" port: 40767 } }
I20240502 21:27:15.343876 6474 leader_election.cc:302] T 00000000000000000000000000000000 P 16ed06c661f4400891d60a84f376a981 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16ed06c661f4400891d60a84f376a981; no voters:
I20240502 21:27:15.344460 6474 leader_election.cc:288] T 00000000000000000000000000000000 P 16ed06c661f4400891d60a84f376a981 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:27:15.344552 6479 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16ed06c661f4400891d60a84f376a981 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:15.345145 6479 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16ed06c661f4400891d60a84f376a981 [term 1 LEADER]: Becoming Leader. State: Replica: 16ed06c661f4400891d60a84f376a981, State: Running, Role: LEADER
I20240502 21:27:15.345651 6479 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16ed06c661f4400891d60a84f376a981 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ed06c661f4400891d60a84f376a981" member_type: VOTER last_known_addr { host: "127.24.226.190" port: 40767 } }
I20240502 21:27:15.346427 6474 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16ed06c661f4400891d60a84f376a981 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:27:15.348150 6480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16ed06c661f4400891d60a84f376a981 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16ed06c661f4400891d60a84f376a981. Latest consensus state: current_term: 1 leader_uuid: "16ed06c661f4400891d60a84f376a981" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ed06c661f4400891d60a84f376a981" member_type: VOTER last_known_addr { host: "127.24.226.190" port: 40767 } } }
I20240502 21:27:15.348451 6480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16ed06c661f4400891d60a84f376a981 [sys.catalog]: This master's current role is: LEADER
I20240502 21:27:15.349030 6481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16ed06c661f4400891d60a84f376a981 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16ed06c661f4400891d60a84f376a981" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ed06c661f4400891d60a84f376a981" member_type: VOTER last_known_addr { host: "127.24.226.190" port: 40767 } } }
I20240502 21:27:15.349221 6481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16ed06c661f4400891d60a84f376a981 [sys.catalog]: This master's current role is: LEADER
I20240502 21:27:15.353763 6485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:27:15.358587 6485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:27:15.366519 6485 catalog_manager.cc:1347] Generated new cluster ID: c0b615aa8bb94d15afee668af33f6e03
I20240502 21:27:15.366618 6485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:27:15.383113 6485 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:27:15.384200 6485 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:27:15.395350 6485 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 16ed06c661f4400891d60a84f376a981: Generated new TSK 0
I20240502 21:27:15.396039 6485 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:27:15.399652 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.129:0
--local_ip_for_outbound_sockets=127.24.226.129
--webserver_interface=127.24.226.129
--webserver_port=0
--tserver_master_addrs=127.24.226.190:40767
--builtin_ntp_servers=127.24.226.148:46251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 21:27:15.720451 6498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:15.720721 6498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:27:15.720811 6498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:15.727095 6498 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:27:15.727180 6498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:27:15.727265 6498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.129
I20240502 21:27:15.734934 6498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:46251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.24.226.129
--webserver_port=0
--tserver_master_addrs=127.24.226.190:40767
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.129
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:27:15.735965 6498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:27:15.737571 6498 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:15.747502 6503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:15.747556 6504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:15.747591 6506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:15.748518 6498 server_base.cc:1034] running on GCE node
I20240502 21:27:15.748992 6498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:27:15.749720 6498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:27:15.750895 6498 hybrid_clock.cc:648] HybridClock initialized: now 1714685235750863 us; error 24 us; skew 500 ppm
I20240502 21:27:15.751152 6498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:15.752982 6498 webserver.cc:460] Webserver started at http://127.24.226.129:40861/ using document root <none> and password file <none>
I20240502 21:27:15.753618 6498 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:15.753705 6498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:15.753960 6498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:27:15.756136 6498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "51bc1a0939fe49348a18d3da18f9feae"
format_stamp: "Formatted at 2024-05-02 21:27:15 on dist-test-slave-41k1"
I20240502 21:27:15.756880 6498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "51bc1a0939fe49348a18d3da18f9feae"
format_stamp: "Formatted at 2024-05-02 21:27:15 on dist-test-slave-41k1"
I20240502 21:27:15.761029 6498 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 21:27:15.763794 6512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:15.764847 6498 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:27:15.764968 6498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "51bc1a0939fe49348a18d3da18f9feae"
format_stamp: "Formatted at 2024-05-02 21:27:15 on dist-test-slave-41k1"
I20240502 21:27:15.765120 6498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:15.789346 6498 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:15.789983 6498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:27:15.790220 6498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:15.790977 6498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:15.792258 6498 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:15.792330 6498 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:15.792459 6498 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:15.792533 6498 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20240502 21:27:15.829835 6498 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.129:38703
I20240502 21:27:15.829933 6624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.129:38703 every 8 connection(s)
I20240502 21:27:15.831310 6498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 21:27:15.832406 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 6498
I20240502 21:27:15.832612 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 21:27:15.836673 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.130:0
--local_ip_for_outbound_sockets=127.24.226.130
--webserver_interface=127.24.226.130
--webserver_port=0
--tserver_master_addrs=127.24.226.190:40767
--builtin_ntp_servers=127.24.226.148:46251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:27:15.847625 6625 heartbeater.cc:347] Connected to a master server at 127.24.226.190:40767
I20240502 21:27:15.847888 6625 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:15.848812 6625 heartbeater.cc:511] Master 127.24.226.190:40767 requested a full tablet report, sending...
I20240502 21:27:15.853946 6439 ts_manager.cc:194] Registered new tserver with Master: 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:15.855393 6439 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.129:52001
W20240502 21:27:16.137320 6629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:16.137585 6629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:27:16.137714 6629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:16.144050 6629 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:27:16.144294 6629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:27:16.144471 6629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.130
I20240502 21:27:16.152112 6629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:46251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.24.226.130
--webserver_port=0
--tserver_master_addrs=127.24.226.190:40767
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.130
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:27:16.153131 6629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:27:16.154639 6629 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:16.164852 6637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:16.164927 6635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:16.164952 6634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:16.165867 6629 server_base.cc:1034] running on GCE node
I20240502 21:27:16.166405 6629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:27:16.167192 6629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:27:16.168416 6629 hybrid_clock.cc:648] HybridClock initialized: now 1714685236168378 us; error 45 us; skew 500 ppm
I20240502 21:27:16.168723 6629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:16.171053 6629 webserver.cc:460] Webserver started at http://127.24.226.130:39495/ using document root <none> and password file <none>
I20240502 21:27:16.171690 6629 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:16.171775 6629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:16.172042 6629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:27:16.174266 6629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
format_stamp: "Formatted at 2024-05-02 21:27:16 on dist-test-slave-41k1"
I20240502 21:27:16.174891 6629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
format_stamp: "Formatted at 2024-05-02 21:27:16 on dist-test-slave-41k1"
I20240502 21:27:16.179719 6629 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20240502 21:27:16.182651 6643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:16.183699 6629 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:16.183830 6629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
format_stamp: "Formatted at 2024-05-02 21:27:16 on dist-test-slave-41k1"
I20240502 21:27:16.184002 6629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:16.200366 6629 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:16.201009 6629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:27:16.201236 6629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:16.202030 6629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:16.203279 6629 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:16.203368 6629 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:16.203473 6629 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:16.203541 6629 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:16.229768 6629 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.130:42515
I20240502 21:27:16.229828 6755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.130:42515 every 8 connection(s)
I20240502 21:27:16.230832 6629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 21:27:16.239713 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 6629
I20240502 21:27:16.239902 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 21:27:16.241382 6756 heartbeater.cc:347] Connected to a master server at 127.24.226.190:40767
I20240502 21:27:16.241613 6756 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:16.242457 6756 heartbeater.cc:511] Master 127.24.226.190:40767 requested a full tablet report, sending...
I20240502 21:27:16.242946 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.131:0
--local_ip_for_outbound_sockets=127.24.226.131
--webserver_interface=127.24.226.131
--webserver_port=0
--tserver_master_addrs=127.24.226.190:40767
--builtin_ntp_servers=127.24.226.148:46251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:27:16.243690 6439 ts_manager.cc:194] Registered new tserver with Master: ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:16.244320 6439 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.130:42285
W20240502 21:27:16.542692 6759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:16.542950 6759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:27:16.543102 6759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:16.549758 6759 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:27:16.549870 6759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:27:16.550024 6759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.131
I20240502 21:27:16.557811 6759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:46251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.24.226.131
--webserver_port=0
--tserver_master_addrs=127.24.226.190:40767
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.131
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:27:16.558825 6759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:27:16.560286 6759 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:16.570901 6764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:16.570977 6767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:16.570999 6765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:16.571686 6759 server_base.cc:1034] running on GCE node
I20240502 21:27:16.572170 6759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:27:16.572994 6759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:27:16.574183 6759 hybrid_clock.cc:648] HybridClock initialized: now 1714685236574152 us; error 21 us; skew 500 ppm
I20240502 21:27:16.574443 6759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:16.576397 6759 webserver.cc:460] Webserver started at http://127.24.226.131:39339/ using document root <none> and password file <none>
I20240502 21:27:16.577078 6759 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:16.577163 6759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:16.577410 6759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:27:16.579512 6759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "9793130a8e4f45388357b8d2e9f47720"
format_stamp: "Formatted at 2024-05-02 21:27:16 on dist-test-slave-41k1"
I20240502 21:27:16.580112 6759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "9793130a8e4f45388357b8d2e9f47720"
format_stamp: "Formatted at 2024-05-02 21:27:16 on dist-test-slave-41k1"
I20240502 21:27:16.584576 6759 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20240502 21:27:16.587574 6773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:16.588732 6759 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:27:16.588897 6759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9793130a8e4f45388357b8d2e9f47720"
format_stamp: "Formatted at 2024-05-02 21:27:16 on dist-test-slave-41k1"
I20240502 21:27:16.589064 6759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:16.605386 6759 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:16.606029 6759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:27:16.606256 6759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:16.607064 6759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:16.608292 6759 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:16.608371 6759 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:16.608454 6759 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:16.608522 6759 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20240502 21:27:16.637324 6759 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.131:33929
I20240502 21:27:16.637392 6885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.131:33929 every 8 connection(s)
I20240502 21:27:16.638361 6759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 21:27:16.641347 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 6759
I20240502 21:27:16.641531 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 21:27:16.644562 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.132:0
--local_ip_for_outbound_sockets=127.24.226.132
--webserver_interface=127.24.226.132
--webserver_port=0
--tserver_master_addrs=127.24.226.190:40767
--builtin_ntp_servers=127.24.226.148:46251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:27:16.650075 6886 heartbeater.cc:347] Connected to a master server at 127.24.226.190:40767
I20240502 21:27:16.650314 6886 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:16.651007 6886 heartbeater.cc:511] Master 127.24.226.190:40767 requested a full tablet report, sending...
I20240502 21:27:16.652381 6439 ts_manager.cc:194] Registered new tserver with Master: 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:16.653163 6439 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.131:57755
I20240502 21:27:16.858592 6625 heartbeater.cc:503] Master 127.24.226.190:40767 was elected leader, sending a full tablet report...
W20240502 21:27:16.946080 6889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:16.946321 6889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:27:16.946451 6889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:16.953622 6889 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:27:16.953719 6889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:27:16.953853 6889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.132
I20240502 21:27:16.961686 6889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:46251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.24.226.132
--webserver_port=0
--tserver_master_addrs=127.24.226.190:40767
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.132
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:27:16.962826 6889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:27:16.964290 6889 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:16.974753 6897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:16.974735 6895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:16.975044 6894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:16.975483 6889 server_base.cc:1034] running on GCE node
I20240502 21:27:16.976037 6889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:27:16.976955 6889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:27:16.978144 6889 hybrid_clock.cc:648] HybridClock initialized: now 1714685236978112 us; error 39 us; skew 500 ppm
I20240502 21:27:16.978417 6889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:16.980324 6889 webserver.cc:460] Webserver started at http://127.24.226.132:41369/ using document root <none> and password file <none>
I20240502 21:27:16.980952 6889 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:16.981065 6889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:16.981346 6889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:27:16.983489 6889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "80a838c00b544bf68ec4cc9c513212b2"
format_stamp: "Formatted at 2024-05-02 21:27:16 on dist-test-slave-41k1"
I20240502 21:27:16.984159 6889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "80a838c00b544bf68ec4cc9c513212b2"
format_stamp: "Formatted at 2024-05-02 21:27:16 on dist-test-slave-41k1"
I20240502 21:27:16.988716 6889 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 21:27:16.991564 6903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:16.992727 6889 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:27:16.992858 6889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "80a838c00b544bf68ec4cc9c513212b2"
format_stamp: "Formatted at 2024-05-02 21:27:16 on dist-test-slave-41k1"
I20240502 21:27:16.993019 6889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:17.021466 6889 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:17.022140 6889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:27:17.022390 6889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:17.023190 6889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:17.024502 6889 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:17.024595 6889 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:17.024703 6889 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:17.024771 6889 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:17.053843 6889 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.132:43031
I20240502 21:27:17.053901 7015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.132:43031 every 8 connection(s)
I20240502 21:27:17.055442 6889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 21:27:17.065737 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 6889
I20240502 21:27:17.065937 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 21:27:17.066658 7016 heartbeater.cc:347] Connected to a master server at 127.24.226.190:40767
I20240502 21:27:17.066928 7016 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:17.067687 7016 heartbeater.cc:511] Master 127.24.226.190:40767 requested a full tablet report, sending...
I20240502 21:27:17.069000 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.133:0
--local_ip_for_outbound_sockets=127.24.226.133
--webserver_interface=127.24.226.133
--webserver_port=0
--tserver_master_addrs=127.24.226.190:40767
--builtin_ntp_servers=127.24.226.148:46251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:27:17.069032 6439 ts_manager.cc:194] Registered new tserver with Master: 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:17.069813 6439 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.132:50509
I20240502 21:27:17.246347 6756 heartbeater.cc:503] Master 127.24.226.190:40767 was elected leader, sending a full tablet report...
W20240502 21:27:17.366870 7019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:17.367123 7019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:27:17.367244 7019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:17.373848 7019 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:27:17.373965 7019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:27:17.374096 7019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.133
I20240502 21:27:17.381922 7019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:46251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.24.226.133
--webserver_port=0
--tserver_master_addrs=127.24.226.190:40767
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.133
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:27:17.382970 7019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:27:17.384451 7019 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:17.395323 7027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:17.395332 7024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:17.395377 7025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:17.396349 7019 server_base.cc:1034] running on GCE node
I20240502 21:27:17.396921 7019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:27:17.397729 7019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:27:17.398931 7019 hybrid_clock.cc:648] HybridClock initialized: now 1714685237398893 us; error 38 us; skew 500 ppm
I20240502 21:27:17.399223 7019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:17.406097 7019 webserver.cc:460] Webserver started at http://127.24.226.133:39743/ using document root <none> and password file <none>
I20240502 21:27:17.406775 7019 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:17.406903 7019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:17.407236 7019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:27:17.409536 7019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "6676d476ceb44beba1c2de8395f4627d"
format_stamp: "Formatted at 2024-05-02 21:27:17 on dist-test-slave-41k1"
I20240502 21:27:17.410306 7019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "6676d476ceb44beba1c2de8395f4627d"
format_stamp: "Formatted at 2024-05-02 21:27:17 on dist-test-slave-41k1"
I20240502 21:27:17.414702 7019 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 21:27:17.417685 7033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:17.418732 7019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:27:17.418870 7019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "6676d476ceb44beba1c2de8395f4627d"
format_stamp: "Formatted at 2024-05-02 21:27:17 on dist-test-slave-41k1"
I20240502 21:27:17.419045 7019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:17.448027 7019 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:17.448745 7019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:27:17.449012 7019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:17.449855 7019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:17.451328 7019 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:17.451431 7019 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:17.451542 7019 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:17.451604 7019 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:17.481009 7019 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.133:42877
I20240502 21:27:17.481065 7145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.133:42877 every 8 connection(s)
I20240502 21:27:17.482127 7019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:27:17.487933 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 7019
I20240502 21:27:17.488080 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 21:27:17.490815 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.134:0
--local_ip_for_outbound_sockets=127.24.226.134
--webserver_interface=127.24.226.134
--webserver_port=0
--tserver_master_addrs=127.24.226.190:40767
--builtin_ntp_servers=127.24.226.148:46251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:27:17.493476 7146 heartbeater.cc:347] Connected to a master server at 127.24.226.190:40767
I20240502 21:27:17.493781 7146 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:17.494683 7146 heartbeater.cc:511] Master 127.24.226.190:40767 requested a full tablet report, sending...
I20240502 21:27:17.496129 6439 ts_manager.cc:194] Registered new tserver with Master: 6676d476ceb44beba1c2de8395f4627d (127.24.226.133:42877)
I20240502 21:27:17.497254 6439 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.133:43839
I20240502 21:27:17.655198 6886 heartbeater.cc:503] Master 127.24.226.190:40767 was elected leader, sending a full tablet report...
W20240502 21:27:17.792357 7149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:17.792703 7149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:27:17.792845 7149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:17.799983 7149 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:27:17.800103 7149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:27:17.800244 7149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.134
I20240502 21:27:17.808892 7149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:46251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.24.226.134
--webserver_port=0
--tserver_master_addrs=127.24.226.190:40767
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.134
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:27:17.810129 7149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:27:17.811744 7149 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:17.824056 7155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:17.824283 7157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:17.824379 7154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:17.825179 7149 server_base.cc:1034] running on GCE node
I20240502 21:27:17.825724 7149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:27:17.826581 7149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:27:17.827771 7149 hybrid_clock.cc:648] HybridClock initialized: now 1714685237827726 us; error 37 us; skew 500 ppm
I20240502 21:27:17.828054 7149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:17.831398 7149 webserver.cc:460] Webserver started at http://127.24.226.134:44171/ using document root <none> and password file <none>
I20240502 21:27:17.832070 7149 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:17.832185 7149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:17.832527 7149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:27:17.835016 7149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "e329c23b07f34d4597def2ad455b9c0c"
format_stamp: "Formatted at 2024-05-02 21:27:17 on dist-test-slave-41k1"
I20240502 21:27:17.835738 7149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "e329c23b07f34d4597def2ad455b9c0c"
format_stamp: "Formatted at 2024-05-02 21:27:17 on dist-test-slave-41k1"
I20240502 21:27:17.840581 7149 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.002s
I20240502 21:27:17.843509 7163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:17.844671 7149 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:27:17.844816 7149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "e329c23b07f34d4597def2ad455b9c0c"
format_stamp: "Formatted at 2024-05-02 21:27:17 on dist-test-slave-41k1"
I20240502 21:27:17.844988 7149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:17.861145 7149 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:17.861829 7149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:27:17.862108 7149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:17.862962 7149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:17.864265 7149 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:17.864362 7149 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:17.864480 7149 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:17.864550 7149 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:17.894296 7149 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.134:36267
I20240502 21:27:17.894389 7277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.134:36267 every 8 connection(s)
I20240502 21:27:17.895438 7149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 21:27:17.901604 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 7149
I20240502 21:27:17.901759 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 21:27:17.904897 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.135:0
--local_ip_for_outbound_sockets=127.24.226.135
--webserver_interface=127.24.226.135
--webserver_port=0
--tserver_master_addrs=127.24.226.190:40767
--builtin_ntp_servers=127.24.226.148:46251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:27:17.907191 7278 heartbeater.cc:347] Connected to a master server at 127.24.226.190:40767
I20240502 21:27:17.907445 7278 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:17.908160 7278 heartbeater.cc:511] Master 127.24.226.190:40767 requested a full tablet report, sending...
I20240502 21:27:17.909551 6439 ts_manager.cc:194] Registered new tserver with Master: e329c23b07f34d4597def2ad455b9c0c (127.24.226.134:36267)
I20240502 21:27:17.910616 6439 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.134:45149
I20240502 21:27:18.072023 7016 heartbeater.cc:503] Master 127.24.226.190:40767 was elected leader, sending a full tablet report...
W20240502 21:27:18.214447 7281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:18.214692 7281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:27:18.214833 7281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:18.222529 7281 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:27:18.222630 7281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:27:18.222731 7281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.135
I20240502 21:27:18.230566 7281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:46251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.24.226.135
--webserver_port=0
--tserver_master_addrs=127.24.226.190:40767
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.135
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:27:18.231591 7281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:27:18.233130 7281 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:18.243491 7286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:18.243515 7287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:18.243711 7289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:18.244295 7281 server_base.cc:1034] running on GCE node
I20240502 21:27:18.244804 7281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:27:18.245553 7281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:27:18.246757 7281 hybrid_clock.cc:648] HybridClock initialized: now 1714685238246705 us; error 46 us; skew 500 ppm
I20240502 21:27:18.247063 7281 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:18.249095 7281 webserver.cc:460] Webserver started at http://127.24.226.135:33877/ using document root <none> and password file <none>
I20240502 21:27:18.249897 7281 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:18.250013 7281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:18.250313 7281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:27:18.252517 7281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "34f4e21e024e464b9450592a5d9a6c8b"
format_stamp: "Formatted at 2024-05-02 21:27:18 on dist-test-slave-41k1"
I20240502 21:27:18.253161 7281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "34f4e21e024e464b9450592a5d9a6c8b"
format_stamp: "Formatted at 2024-05-02 21:27:18 on dist-test-slave-41k1"
I20240502 21:27:18.257635 7281 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 21:27:18.260401 7295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:18.261445 7281 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 21:27:18.261569 7281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "34f4e21e024e464b9450592a5d9a6c8b"
format_stamp: "Formatted at 2024-05-02 21:27:18 on dist-test-slave-41k1"
I20240502 21:27:18.261727 7281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:18.274880 7281 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:18.275535 7281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:27:18.275776 7281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:18.276607 7281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:18.277917 7281 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:18.277990 7281 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:18.278113 7281 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:18.278184 7281 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:18.310618 7281 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.135:43095
I20240502 21:27:18.310675 7408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.135:43095 every 8 connection(s)
I20240502 21:27:18.311718 7281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 21:27:18.315451 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 7281
I20240502 21:27:18.315629 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 21:27:18.318624 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.136:0
--local_ip_for_outbound_sockets=127.24.226.136
--webserver_interface=127.24.226.136
--webserver_port=0
--tserver_master_addrs=127.24.226.190:40767
--builtin_ntp_servers=127.24.226.148:46251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:27:18.325104 7409 heartbeater.cc:347] Connected to a master server at 127.24.226.190:40767
I20240502 21:27:18.325369 7409 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:18.326102 7409 heartbeater.cc:511] Master 127.24.226.190:40767 requested a full tablet report, sending...
I20240502 21:27:18.327343 6439 ts_manager.cc:194] Registered new tserver with Master: 34f4e21e024e464b9450592a5d9a6c8b (127.24.226.135:43095)
I20240502 21:27:18.328024 6439 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.135:55053
I20240502 21:27:18.499603 7146 heartbeater.cc:503] Master 127.24.226.190:40767 was elected leader, sending a full tablet report...
W20240502 21:27:18.613277 7412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:18.613528 7412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:27:18.613662 7412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:18.620168 7412 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:27:18.620285 7412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:27:18.620426 7412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.136
I20240502 21:27:18.628368 7412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:46251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.24.226.136
--webserver_port=0
--tserver_master_addrs=127.24.226.190:40767
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.136
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:27:18.629479 7412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:27:18.630939 7412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:18.642256 7417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:18.642263 7420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:18.642285 7418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:18.642887 7412 server_base.cc:1034] running on GCE node
I20240502 21:27:18.643373 7412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:27:18.644119 7412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:27:18.645292 7412 hybrid_clock.cc:648] HybridClock initialized: now 1714685238645252 us; error 36 us; skew 500 ppm
I20240502 21:27:18.645553 7412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:18.647634 7412 webserver.cc:460] Webserver started at http://127.24.226.136:46577/ using document root <none> and password file <none>
I20240502 21:27:18.648253 7412 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:18.648358 7412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:18.648654 7412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:27:18.650985 7412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "8a9bbbab70e444c18da6ad8f7a2c290a"
format_stamp: "Formatted at 2024-05-02 21:27:18 on dist-test-slave-41k1"
I20240502 21:27:18.651640 7412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "8a9bbbab70e444c18da6ad8f7a2c290a"
format_stamp: "Formatted at 2024-05-02 21:27:18 on dist-test-slave-41k1"
I20240502 21:27:18.656327 7412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20240502 21:27:18.659492 7426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:18.660669 7412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20240502 21:27:18.660806 7412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "8a9bbbab70e444c18da6ad8f7a2c290a"
format_stamp: "Formatted at 2024-05-02 21:27:18 on dist-test-slave-41k1"
I20240502 21:27:18.660967 7412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:18.675849 7412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:18.676626 7412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:27:18.676872 7412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:18.677749 7412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:18.678997 7412 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:18.679090 7412 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:18.679193 7412 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:18.679256 7412 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:18.711833 7412 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.136:45511
I20240502 21:27:18.711884 7539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.136:45511 every 8 connection(s)
I20240502 21:27:18.713104 7412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 21:27:18.718811 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 7412
I20240502 21:27:18.719007 25482 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 21:27:18.725204 7540 heartbeater.cc:347] Connected to a master server at 127.24.226.190:40767
I20240502 21:27:18.725435 7540 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:18.726168 7540 heartbeater.cc:511] Master 127.24.226.190:40767 requested a full tablet report, sending...
I20240502 21:27:18.727447 6439 ts_manager.cc:194] Registered new tserver with Master: 8a9bbbab70e444c18da6ad8f7a2c290a (127.24.226.136:45511)
I20240502 21:27:18.728170 6439 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.136:51829
I20240502 21:27:18.734090 25482 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 21:27:18.748597 25482 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 21:27:18.748721 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 7019
I20240502 21:27:18.755875 7141 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:27:18.832233 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 7019
I20240502 21:27:18.849880 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 7149
I20240502 21:27:18.857026 7273 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:27:18.928673 7278 heartbeater.cc:503] Master 127.24.226.190:40767 was elected leader, sending a full tablet report...
I20240502 21:27:18.932561 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 7149
I20240502 21:27:18.949194 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 7281
I20240502 21:27:18.956110 7404 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:27:19.030189 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 7281
I20240502 21:27:19.046784 25482 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskD89o3N/build/asan/bin/kudu pid 7412
I20240502 21:27:19.055029 7535 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:27:19.128554 25482 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD89o3N/build/asan/bin/kudu with pid 7412
I20240502 21:27:22.895738 6439 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48454:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:22.937440 6691 tablet_service.cc:1467] Processing CreateTablet for tablet 1325bd7bbb4340f88bdc7c705934862e (DEFAULT_TABLE table=rebalance_test_table_0 [id=4065ee5c62f94d5e8ab06286ee31af48]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:22.937242 6690 tablet_service.cc:1467] Processing CreateTablet for tablet c6d11cba6ca04c17a9f47506824d9aa2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4065ee5c62f94d5e8ab06286ee31af48]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:22.938735 6690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d11cba6ca04c17a9f47506824d9aa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:22.942736 6691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1325bd7bbb4340f88bdc7c705934862e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:22.948348 6951 tablet_service.cc:1467] Processing CreateTablet for tablet 1325bd7bbb4340f88bdc7c705934862e (DEFAULT_TABLE table=rebalance_test_table_0 [id=4065ee5c62f94d5e8ab06286ee31af48]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:22.949879 6951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1325bd7bbb4340f88bdc7c705934862e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:22.950444 6559 tablet_service.cc:1467] Processing CreateTablet for tablet c6d11cba6ca04c17a9f47506824d9aa2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4065ee5c62f94d5e8ab06286ee31af48]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:22.951355 6560 tablet_service.cc:1467] Processing CreateTablet for tablet 1325bd7bbb4340f88bdc7c705934862e (DEFAULT_TABLE table=rebalance_test_table_0 [id=4065ee5c62f94d5e8ab06286ee31af48]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:22.951673 6950 tablet_service.cc:1467] Processing CreateTablet for tablet 5e38d544ee14490c9fb5d8069440fd6a (DEFAULT_TABLE table=rebalance_test_table_0 [id=4065ee5c62f94d5e8ab06286ee31af48]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:22.951864 6559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d11cba6ca04c17a9f47506824d9aa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:22.952069 6950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e38d544ee14490c9fb5d8069440fd6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:22.952733 6558 tablet_service.cc:1467] Processing CreateTablet for tablet 5e38d544ee14490c9fb5d8069440fd6a (DEFAULT_TABLE table=rebalance_test_table_0 [id=4065ee5c62f94d5e8ab06286ee31af48]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:22.953408 6560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1325bd7bbb4340f88bdc7c705934862e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:22.953619 6558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e38d544ee14490c9fb5d8069440fd6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:22.955806 6821 tablet_service.cc:1467] Processing CreateTablet for tablet c6d11cba6ca04c17a9f47506824d9aa2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4065ee5c62f94d5e8ab06286ee31af48]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:22.957212 6820 tablet_service.cc:1467] Processing CreateTablet for tablet 5e38d544ee14490c9fb5d8069440fd6a (DEFAULT_TABLE table=rebalance_test_table_0 [id=4065ee5c62f94d5e8ab06286ee31af48]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:22.957686 6821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d11cba6ca04c17a9f47506824d9aa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:22.958104 6820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e38d544ee14490c9fb5d8069440fd6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:22.971557 7571 tablet_bootstrap.cc:492] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae: Bootstrap starting.
I20240502 21:27:22.973292 7570 tablet_bootstrap.cc:492] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef: Bootstrap starting.
I20240502 21:27:22.973429 7573 tablet_bootstrap.cc:492] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720: Bootstrap starting.
I20240502 21:27:22.975265 7571 tablet_bootstrap.cc:654] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:22.976985 7571 log.cc:824] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae: Log is configured to *not* fsync() on all Append() calls
I20240502 21:27:22.978122 7572 tablet_bootstrap.cc:492] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2: Bootstrap starting.
I20240502 21:27:22.978240 7570 tablet_bootstrap.cc:654] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:22.978566 7573 tablet_bootstrap.cc:654] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:22.979651 7573 log.cc:824] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720: Log is configured to *not* fsync() on all Append() calls
I20240502 21:27:22.979719 7570 log.cc:824] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef: Log is configured to *not* fsync() on all Append() calls
I20240502 21:27:22.981377 7572 tablet_bootstrap.cc:654] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:22.981721 7571 tablet_bootstrap.cc:492] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae: No bootstrap required, opened a new log
I20240502 21:27:22.981962 7571 ts_tablet_manager.cc:1367] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20240502 21:27:22.982098 7570 tablet_bootstrap.cc:492] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef: No bootstrap required, opened a new log
I20240502 21:27:22.982309 7570 ts_tablet_manager.cc:1367] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20240502 21:27:22.982410 7572 log.cc:824] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2: Log is configured to *not* fsync() on all Append() calls
I20240502 21:27:22.984385 7573 tablet_bootstrap.cc:492] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720: No bootstrap required, opened a new log
I20240502 21:27:22.984644 7573 ts_tablet_manager.cc:1367] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.001s
I20240502 21:27:22.988313 7570 raft_consensus.cc:348] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:22.988941 7572 tablet_bootstrap.cc:492] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2: No bootstrap required, opened a new log
I20240502 21:27:22.988950 7570 raft_consensus.cc:374] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:22.989099 7570 raft_consensus.cc:729] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Initialized, Role: FOLLOWER
I20240502 21:27:22.989233 7572 ts_tablet_manager.cc:1367] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.002s
I20240502 21:27:22.989897 7570 consensus_queue.cc:260] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:22.990180 7571 raft_consensus.cc:348] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:22.990577 7571 raft_consensus.cc:374] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:22.990695 7571 raft_consensus.cc:729] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Initialized, Role: FOLLOWER
I20240502 21:27:22.991370 7571 consensus_queue.cc:260] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:22.992707 7573 raft_consensus.cc:348] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:22.993232 7573 raft_consensus.cc:374] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:22.993397 7573 raft_consensus.cc:729] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Initialized, Role: FOLLOWER
I20240502 21:27:22.994459 7573 consensus_queue.cc:260] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:22.999480 7571 ts_tablet_manager.cc:1398] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae: Time spent starting tablet: real 0.017s user 0.011s sys 0.006s
I20240502 21:27:22.999933 7571 tablet_bootstrap.cc:492] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae: Bootstrap starting.
I20240502 21:27:23.000414 7578 raft_consensus.cc:482] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:23.000605 7572 raft_consensus.cc:348] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:23.001042 7572 raft_consensus.cc:374] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:23.001176 7572 raft_consensus.cc:729] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Initialized, Role: FOLLOWER
I20240502 21:27:23.001543 7571 tablet_bootstrap.cc:654] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:23.001899 7572 consensus_queue.cc:260] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:23.004529 7578 raft_consensus.cc:504] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:23.005539 7573 ts_tablet_manager.cc:1398] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720: Time spent starting tablet: real 0.021s user 0.005s sys 0.008s
I20240502 21:27:23.004299 7570 ts_tablet_manager.cc:1398] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent starting tablet: real 0.022s user 0.007s sys 0.009s
I20240502 21:27:23.006160 7570 tablet_bootstrap.cc:492] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef: Bootstrap starting.
I20240502 21:27:23.007148 7573 tablet_bootstrap.cc:492] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720: Bootstrap starting.
I20240502 21:27:23.009480 7573 tablet_bootstrap.cc:654] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:23.009651 7571 tablet_bootstrap.cc:492] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae: No bootstrap required, opened a new log
I20240502 21:27:23.009634 7572 ts_tablet_manager.cc:1398] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2: Time spent starting tablet: real 0.020s user 0.007s sys 0.010s
I20240502 21:27:23.009830 7571 ts_tablet_manager.cc:1367] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240502 21:27:23.010900 7571 raft_consensus.cc:348] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:23.011219 7571 raft_consensus.cc:374] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:23.011366 7572 tablet_bootstrap.cc:492] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2: Bootstrap starting.
I20240502 21:27:23.011359 7571 raft_consensus.cc:729] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Initialized, Role: FOLLOWER
I20240502 21:27:23.011670 7571 consensus_queue.cc:260] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:23.012445 7571 ts_tablet_manager.cc:1398] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:27:23.012775 7571 tablet_bootstrap.cc:492] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae: Bootstrap starting.
I20240502 21:27:23.012888 7572 tablet_bootstrap.cc:654] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:23.014694 7571 tablet_bootstrap.cc:654] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:23.019445 7570 tablet_bootstrap.cc:654] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:23.020139 7572 tablet_bootstrap.cc:492] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2: No bootstrap required, opened a new log
I20240502 21:27:23.020278 7571 tablet_bootstrap.cc:492] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae: No bootstrap required, opened a new log
I20240502 21:27:23.020285 7572 ts_tablet_manager.cc:1367] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240502 21:27:23.020367 7573 tablet_bootstrap.cc:492] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720: No bootstrap required, opened a new log
I20240502 21:27:23.020505 7571 ts_tablet_manager.cc:1367] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240502 21:27:23.020560 7573 ts_tablet_manager.cc:1367] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.010s
I20240502 21:27:23.021169 7572 raft_consensus.cc:348] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:23.021209 7571 raft_consensus.cc:348] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:23.021493 7572 raft_consensus.cc:374] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:23.021592 7572 raft_consensus.cc:729] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Initialized, Role: FOLLOWER
I20240502 21:27:23.021598 7571 raft_consensus.cc:374] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:23.021693 7571 raft_consensus.cc:729] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Initialized, Role: FOLLOWER
I20240502 21:27:23.021953 7571 consensus_queue.cc:260] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:23.021858 7572 consensus_queue.cc:260] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:23.022866 7571 ts_tablet_manager.cc:1398] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:23.023290 7572 ts_tablet_manager.cc:1398] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:27:23.023567 7573 raft_consensus.cc:348] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:23.023891 7573 raft_consensus.cc:374] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:23.024039 7573 raft_consensus.cc:729] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Initialized, Role: FOLLOWER
I20240502 21:27:23.024336 7573 consensus_queue.cc:260] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:23.025039 7573 ts_tablet_manager.cc:1398] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:23.025976 7578 leader_election.cc:288] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:23.027418 7570 tablet_bootstrap.cc:492] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef: No bootstrap required, opened a new log
I20240502 21:27:23.027649 7570 ts_tablet_manager.cc:1367] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.002s
I20240502 21:27:23.028692 7570 raft_consensus.cc:348] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:23.028995 7570 raft_consensus.cc:374] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:23.029098 7570 raft_consensus.cc:729] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Initialized, Role: FOLLOWER
I20240502 21:27:23.029361 7570 consensus_queue.cc:260] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:23.029798 7570 ts_tablet_manager.cc:1398] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:23.032604 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1325bd7bbb4340f88bdc7c705934862e" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bc1a0939fe49348a18d3da18f9feae" is_pre_election: true
I20240502 21:27:23.033125 6580 raft_consensus.cc:2454] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 0.
I20240502 21:27:23.033315 6971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1325bd7bbb4340f88bdc7c705934862e" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a838c00b544bf68ec4cc9c513212b2" is_pre_election: true
I20240502 21:27:23.033725 6644 leader_election.cc:302] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:23.033963 6971 raft_consensus.cc:2454] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 0.
I20240502 21:27:23.034044 7578 raft_consensus.cc:2781] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:23.034193 7578 raft_consensus.cc:482] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:23.034307 7578 raft_consensus.cc:3037] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:23.036278 7578 raft_consensus.cc:504] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:23.037230 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1325bd7bbb4340f88bdc7c705934862e" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:23.037446 6580 raft_consensus.cc:3037] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:23.037467 6971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1325bd7bbb4340f88bdc7c705934862e" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:23.037633 6971 raft_consensus.cc:3037] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:23.039461 6580 raft_consensus.cc:2454] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 1.
I20240502 21:27:23.039686 7578 leader_election.cc:288] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 1 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:23.039984 6644 leader_election.cc:302] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:23.039969 6971 raft_consensus.cc:2454] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 1.
I20240502 21:27:23.040493 7578 raft_consensus.cc:2781] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:23.040674 7578 raft_consensus.cc:686] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:23.041153 7578 consensus_queue.cc:237] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:23.045661 6439 catalog_manager.cc:5484] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 0 to 1, leader changed from <none> to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 1 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:23.056576 7581 raft_consensus.cc:482] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:23.056792 7581 raft_consensus.cc:504] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:23.058393 7581 leader_election.cc:288] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
W20240502 21:27:23.060606 7017 tablet.cc:2369] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:27:23.066948 6971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e38d544ee14490c9fb5d8069440fd6a" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a838c00b544bf68ec4cc9c513212b2" is_pre_election: true
I20240502 21:27:23.066955 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e38d544ee14490c9fb5d8069440fd6a" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bc1a0939fe49348a18d3da18f9feae" is_pre_election: true
I20240502 21:27:23.067162 6971 raft_consensus.cc:2454] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 0.
I20240502 21:27:23.067175 6580 raft_consensus.cc:2454] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 0.
I20240502 21:27:23.067649 6774 leader_election.cc:302] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:23.067978 7581 raft_consensus.cc:2781] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:23.068135 7581 raft_consensus.cc:482] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:23.068253 7581 raft_consensus.cc:3037] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:23.070390 7581 raft_consensus.cc:504] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:23.071014 7581 leader_election.cc:288] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:23.071336 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e38d544ee14490c9fb5d8069440fd6a" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:23.071517 6580 raft_consensus.cc:3037] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:23.071557 6971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e38d544ee14490c9fb5d8069440fd6a" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:23.071766 6971 raft_consensus.cc:3037] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:23.073359 6580 raft_consensus.cc:2454] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 1.
I20240502 21:27:23.073601 6971 raft_consensus.cc:2454] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 1.
I20240502 21:27:23.073853 6774 leader_election.cc:302] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:23.074191 7581 raft_consensus.cc:2781] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:23.074613 7581 raft_consensus.cc:686] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 1 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:23.075085 7581 consensus_queue.cc:237] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:23.079155 6439 catalog_manager.cc:5484] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 1 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
W20240502 21:27:23.087498 6626 tablet.cc:2369] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 21:27:23.143774 6887 tablet.cc:2369] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 21:27:23.237025 6757 tablet.cc:2369] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:27:23.243060 7581 raft_consensus.cc:482] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:23.243261 7581 raft_consensus.cc:504] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:23.243851 7581 leader_election.cc:288] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:23.244452 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d11cba6ca04c17a9f47506824d9aa2" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bc1a0939fe49348a18d3da18f9feae" is_pre_election: true
I20240502 21:27:23.244680 6580 raft_consensus.cc:2454] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 0.
I20240502 21:27:23.245083 6774 leader_election.cc:302] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:23.245321 7581 raft_consensus.cc:2781] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:23.245436 7581 raft_consensus.cc:482] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:23.245543 7581 raft_consensus.cc:3037] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:23.248047 7581 raft_consensus.cc:504] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:23.248883 7581 leader_election.cc:288] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:23.249017 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d11cba6ca04c17a9f47506824d9aa2" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:23.249186 6580 raft_consensus.cc:3037] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:23.250074 6710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d11cba6ca04c17a9f47506824d9aa2" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:23.250129 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d11cba6ca04c17a9f47506824d9aa2" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" is_pre_election: true
I20240502 21:27:23.250424 6710 raft_consensus.cc:3037] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:23.251575 6580 raft_consensus.cc:2454] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 1.
I20240502 21:27:23.251880 6774 leader_election.cc:302] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:23.252097 7581 raft_consensus.cc:2781] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:23.252238 7581 raft_consensus.cc:686] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 1 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:23.252532 6710 raft_consensus.cc:2454] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 1.
I20240502 21:27:23.252480 7581 consensus_queue.cc:237] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:23.254894 6439 catalog_manager.cc:5484] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 1 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:23.544585 7578 consensus_queue.cc:1035] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:23.552239 7599 consensus_queue.cc:1035] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:23.563560 7581 consensus_queue.cc:1035] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:23.570379 7581 consensus_queue.cc:1035] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:23.594403 7596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:23.594720 7596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:23.601500 7596 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 21:27:23.676133 7607 consensus_queue.cc:1035] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:23.683301 7607 consensus_queue.cc:1035] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 208.699 ms
time per row: 0.10435 ms
I20240502 21:27:23.946388 6439 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48454:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:23.954528 6691 tablet_service.cc:1467] Processing CreateTablet for tablet ea3b52c673ae4b9f931059c4b0a456fa (DEFAULT_TABLE table=rebalance_test_table_1 [id=c33677b3e22b424e859afefd53f4b73c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:23.954530 6950 tablet_service.cc:1467] Processing CreateTablet for tablet ea3b52c673ae4b9f931059c4b0a456fa (DEFAULT_TABLE table=rebalance_test_table_1 [id=c33677b3e22b424e859afefd53f4b73c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:23.954962 6950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea3b52c673ae4b9f931059c4b0a456fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:23.954965 6691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea3b52c673ae4b9f931059c4b0a456fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:23.957233 6821 tablet_service.cc:1467] Processing CreateTablet for tablet ea3b52c673ae4b9f931059c4b0a456fa (DEFAULT_TABLE table=rebalance_test_table_1 [id=c33677b3e22b424e859afefd53f4b73c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:23.957686 6821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea3b52c673ae4b9f931059c4b0a456fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:23.957849 6820 tablet_service.cc:1467] Processing CreateTablet for tablet 57e37f7e15b74819b692792eaed94f24 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c33677b3e22b424e859afefd53f4b73c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:23.958230 6820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57e37f7e15b74819b692792eaed94f24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:23.960464 6558 tablet_service.cc:1467] Processing CreateTablet for tablet f0b941baaab542a78748d4d0894cfbf4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c33677b3e22b424e859afefd53f4b73c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:23.960886 6558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0b941baaab542a78748d4d0894cfbf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:23.960932 6819 tablet_service.cc:1467] Processing CreateTablet for tablet f0b941baaab542a78748d4d0894cfbf4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c33677b3e22b424e859afefd53f4b73c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:23.961283 6819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0b941baaab542a78748d4d0894cfbf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:23.965224 6690 tablet_service.cc:1467] Processing CreateTablet for tablet 57e37f7e15b74819b692792eaed94f24 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c33677b3e22b424e859afefd53f4b73c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:23.965243 6951 tablet_service.cc:1467] Processing CreateTablet for tablet 57e37f7e15b74819b692792eaed94f24 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c33677b3e22b424e859afefd53f4b73c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:23.965348 6689 tablet_service.cc:1467] Processing CreateTablet for tablet f0b941baaab542a78748d4d0894cfbf4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c33677b3e22b424e859afefd53f4b73c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:23.965585 6951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57e37f7e15b74819b692792eaed94f24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:23.965601 6690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57e37f7e15b74819b692792eaed94f24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:23.965768 6689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0b941baaab542a78748d4d0894cfbf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:23.976754 7659 tablet_bootstrap.cc:492] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef: Bootstrap starting.
I20240502 21:27:23.979035 7658 tablet_bootstrap.cc:492] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2: Bootstrap starting.
I20240502 21:27:23.981271 7658 tablet_bootstrap.cc:654] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:23.981922 7660 tablet_bootstrap.cc:492] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae: Bootstrap starting.
I20240502 21:27:23.983067 7661 tablet_bootstrap.cc:492] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720: Bootstrap starting.
I20240502 21:27:23.984032 7660 tablet_bootstrap.cc:654] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:23.984665 7661 tablet_bootstrap.cc:654] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:23.987638 7659 tablet_bootstrap.cc:654] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:23.992539 7660 tablet_bootstrap.cc:492] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae: No bootstrap required, opened a new log
I20240502 21:27:23.992671 7661 tablet_bootstrap.cc:492] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720: No bootstrap required, opened a new log
I20240502 21:27:23.992748 7660 ts_tablet_manager.cc:1367] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240502 21:27:23.992890 7661 ts_tablet_manager.cc:1367] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240502 21:27:23.993817 7660 raft_consensus.cc:348] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:23.993870 7661 raft_consensus.cc:348] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:23.994051 7661 raft_consensus.cc:374] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:23.994164 7661 raft_consensus.cc:729] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Initialized, Role: FOLLOWER
I20240502 21:27:23.994220 7660 raft_consensus.cc:374] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:23.994448 7661 consensus_queue.cc:260] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:23.994743 7660 raft_consensus.cc:729] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Initialized, Role: FOLLOWER
I20240502 21:27:23.994979 7661 ts_tablet_manager.cc:1398] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:23.994997 7660 consensus_queue.cc:260] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:23.995244 7661 tablet_bootstrap.cc:492] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720: Bootstrap starting.
I20240502 21:27:23.995569 7660 ts_tablet_manager.cc:1398] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:23.995920 7658 tablet_bootstrap.cc:492] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2: No bootstrap required, opened a new log
I20240502 21:27:23.996073 7658 ts_tablet_manager.cc:1367] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.014s
I20240502 21:27:23.996162 7659 tablet_bootstrap.cc:492] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef: No bootstrap required, opened a new log
I20240502 21:27:23.996325 7659 ts_tablet_manager.cc:1367] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.007s
I20240502 21:27:23.997159 7659 raft_consensus.cc:348] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:23.997406 7659 raft_consensus.cc:374] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:23.997311 7658 raft_consensus.cc:348] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:23.997501 7659 raft_consensus.cc:729] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Initialized, Role: FOLLOWER
I20240502 21:27:23.997527 7658 raft_consensus.cc:374] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:23.997606 7658 raft_consensus.cc:729] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Initialized, Role: FOLLOWER
I20240502 21:27:23.997699 7659 consensus_queue.cc:260] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:23.997803 7658 consensus_queue.cc:260] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:23.998178 7659 ts_tablet_manager.cc:1398] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:27:23.998396 7659 tablet_bootstrap.cc:492] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef: Bootstrap starting.
I20240502 21:27:23.998968 7658 ts_tablet_manager.cc:1398] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:27:23.999220 7658 tablet_bootstrap.cc:492] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2: Bootstrap starting.
I20240502 21:27:23.999712 7659 tablet_bootstrap.cc:654] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:24.000690 7658 tablet_bootstrap.cc:654] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:24.002177 7659 tablet_bootstrap.cc:492] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef: No bootstrap required, opened a new log
I20240502 21:27:24.002326 7659 ts_tablet_manager.cc:1367] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 21:27:24.003041 7659 raft_consensus.cc:348] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:24.003253 7659 raft_consensus.cc:374] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:24.003340 7659 raft_consensus.cc:729] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Initialized, Role: FOLLOWER
I20240502 21:27:24.003589 7659 consensus_queue.cc:260] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:24.004112 7659 ts_tablet_manager.cc:1398] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:24.004407 7659 tablet_bootstrap.cc:492] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef: Bootstrap starting.
I20240502 21:27:24.005728 7659 tablet_bootstrap.cc:654] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:24.006158 7661 tablet_bootstrap.cc:654] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:24.009152 7658 tablet_bootstrap.cc:492] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2: No bootstrap required, opened a new log
I20240502 21:27:24.009310 7658 ts_tablet_manager.cc:1367] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240502 21:27:24.009375 7661 tablet_bootstrap.cc:492] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720: No bootstrap required, opened a new log
I20240502 21:27:24.009519 7661 ts_tablet_manager.cc:1367] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.001s
I20240502 21:27:24.009991 7658 raft_consensus.cc:348] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:24.010214 7658 raft_consensus.cc:374] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:24.010283 7658 raft_consensus.cc:729] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Initialized, Role: FOLLOWER
I20240502 21:27:24.010159 7661 raft_consensus.cc:348] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:24.010378 7661 raft_consensus.cc:374] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:24.010444 7661 raft_consensus.cc:729] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Initialized, Role: FOLLOWER
I20240502 21:27:24.010504 7658 consensus_queue.cc:260] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:24.010663 7661 consensus_queue.cc:260] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:24.010872 7659 tablet_bootstrap.cc:492] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef: No bootstrap required, opened a new log
I20240502 21:27:24.010986 7659 ts_tablet_manager.cc:1367] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20240502 21:27:24.011225 7661 ts_tablet_manager.cc:1398] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:27:24.011477 7661 tablet_bootstrap.cc:492] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720: Bootstrap starting.
I20240502 21:27:24.011543 7658 ts_tablet_manager.cc:1398] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240502 21:27:24.011605 7659 raft_consensus.cc:348] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:24.011760 7659 raft_consensus.cc:374] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:24.011866 7659 raft_consensus.cc:729] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Initialized, Role: FOLLOWER
I20240502 21:27:24.012066 7659 consensus_queue.cc:260] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:24.012603 7661 tablet_bootstrap.cc:654] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:24.012651 7659 ts_tablet_manager.cc:1398] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:27:24.015684 7661 tablet_bootstrap.cc:492] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720: No bootstrap required, opened a new log
I20240502 21:27:24.015830 7661 ts_tablet_manager.cc:1367] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 21:27:24.016538 7661 raft_consensus.cc:348] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:24.016688 7661 raft_consensus.cc:374] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:24.016753 7661 raft_consensus.cc:729] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Initialized, Role: FOLLOWER
I20240502 21:27:24.016923 7661 consensus_queue.cc:260] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:24.017350 7661 ts_tablet_manager.cc:1398] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 21:27:24.071815 7602 raft_consensus.cc:482] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:24.071967 7602 raft_consensus.cc:504] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:24.073518 7602 leader_election.cc:288] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:24.079928 6710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e37f7e15b74819b692792eaed94f24" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" is_pre_election: true
I20240502 21:27:24.080129 6710 raft_consensus.cc:2454] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 0.
I20240502 21:27:24.080034 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e37f7e15b74819b692792eaed94f24" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9793130a8e4f45388357b8d2e9f47720" is_pre_election: true
I20240502 21:27:24.080471 6841 raft_consensus.cc:2454] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 0.
I20240502 21:27:24.080567 6906 leader_election.cc:302] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:24.080926 7602 raft_consensus.cc:2781] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:24.081053 7602 raft_consensus.cc:482] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:24.081163 7602 raft_consensus.cc:3037] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:24.083004 7602 raft_consensus.cc:504] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:24.083550 7602 leader_election.cc:288] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 1 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:24.083949 6710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e37f7e15b74819b692792eaed94f24" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:24.083999 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e37f7e15b74819b692792eaed94f24" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:24.084146 6710 raft_consensus.cc:3037] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:24.084152 6841 raft_consensus.cc:3037] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:24.085949 6710 raft_consensus.cc:2454] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 1.
I20240502 21:27:24.086015 6841 raft_consensus.cc:2454] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 1.
I20240502 21:27:24.086341 6906 leader_election.cc:302] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:24.086601 7602 raft_consensus.cc:2781] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:24.087026 7602 raft_consensus.cc:686] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:24.087486 7602 consensus_queue.cc:237] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:24.090629 6438 catalog_manager.cc:5484] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 1 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:24.142885 7593 raft_consensus.cc:482] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:24.143014 7593 raft_consensus.cc:504] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:24.143553 7593 leader_election.cc:288] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:24.143985 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b941baaab542a78748d4d0894cfbf4" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bc1a0939fe49348a18d3da18f9feae" is_pre_election: true
I20240502 21:27:24.144012 6710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b941baaab542a78748d4d0894cfbf4" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" is_pre_election: true
I20240502 21:27:24.144178 6710 raft_consensus.cc:2454] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 0.
I20240502 21:27:24.144197 6580 raft_consensus.cc:2454] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 0.
I20240502 21:27:24.144577 6776 leader_election.cc:302] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:24.144824 7593 raft_consensus.cc:2781] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:24.144914 7593 raft_consensus.cc:482] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:24.144991 7593 raft_consensus.cc:3037] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:24.146750 7593 raft_consensus.cc:504] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:24.147336 7593 leader_election.cc:288] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:24.147686 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b941baaab542a78748d4d0894cfbf4" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:24.147737 6710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b941baaab542a78748d4d0894cfbf4" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:24.147842 6580 raft_consensus.cc:3037] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:24.147919 6710 raft_consensus.cc:3037] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:24.149749 6580 raft_consensus.cc:2454] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 1.
I20240502 21:27:24.149752 6710 raft_consensus.cc:2454] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 1.
I20240502 21:27:24.150121 6774 leader_election.cc:302] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:24.150386 7593 raft_consensus.cc:2781] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:24.150516 7593 raft_consensus.cc:686] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 1 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:24.150771 7593 consensus_queue.cc:237] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:24.153038 6438 catalog_manager.cc:5484] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 1 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:24.185756 7593 raft_consensus.cc:482] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:24.185910 7593 raft_consensus.cc:504] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:24.186421 7593 leader_election.cc:288] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:24.186822 6971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3b52c673ae4b9f931059c4b0a456fa" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a838c00b544bf68ec4cc9c513212b2" is_pre_election: true
I20240502 21:27:24.186873 6710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3b52c673ae4b9f931059c4b0a456fa" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" is_pre_election: true
I20240502 21:27:24.187021 6971 raft_consensus.cc:2454] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 0.
I20240502 21:27:24.187065 6710 raft_consensus.cc:2454] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 0.
I20240502 21:27:24.187477 6775 leader_election.cc:302] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:24.187722 7593 raft_consensus.cc:2781] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:24.187808 7593 raft_consensus.cc:482] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:24.187898 7593 raft_consensus.cc:3037] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:24.189554 7593 raft_consensus.cc:504] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:24.190033 7593 leader_election.cc:288] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:24.190407 6971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3b52c673ae4b9f931059c4b0a456fa" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:24.190474 6710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3b52c673ae4b9f931059c4b0a456fa" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:24.190591 6971 raft_consensus.cc:3037] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:24.190659 6710 raft_consensus.cc:3037] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:24.193048 6710 raft_consensus.cc:2454] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 1.
I20240502 21:27:24.193064 6971 raft_consensus.cc:2454] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 1.
I20240502 21:27:24.193395 6775 leader_election.cc:302] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:24.193614 7593 raft_consensus.cc:2781] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:24.193819 7593 raft_consensus.cc:686] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 1 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:24.194041 7593 consensus_queue.cc:237] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:24.196077 6438 catalog_manager.cc:5484] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 1 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:24.499293 7602 consensus_queue.cc:1035] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:24.510753 7673 consensus_queue.cc:1035] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:24.511804 7677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:24.512153 7677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:24.520028 7677 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 21:27:24.624606 7593 consensus_queue.cc:1035] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:24.630944 7581 consensus_queue.cc:1035] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:24.635111 7581 consensus_queue.cc:1035] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:24.662098 7581 consensus_queue.cc:1035] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 240.098 ms
time per row: 0.120049 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 21:27:25.754117 6437 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48454:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:25.762249 6558 tablet_service.cc:1467] Processing CreateTablet for tablet fb023d72aa8d4248a64d4723e18a3cf6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e7dba13b3c454a2ca5a93e5c5cb62935]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:25.762367 6689 tablet_service.cc:1467] Processing CreateTablet for tablet fb023d72aa8d4248a64d4723e18a3cf6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e7dba13b3c454a2ca5a93e5c5cb62935]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:25.762702 6558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb023d72aa8d4248a64d4723e18a3cf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:25.762799 6689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb023d72aa8d4248a64d4723e18a3cf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:25.763514 6950 tablet_service.cc:1467] Processing CreateTablet for tablet fb023d72aa8d4248a64d4723e18a3cf6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e7dba13b3c454a2ca5a93e5c5cb62935]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:25.763644 6951 tablet_service.cc:1467] Processing CreateTablet for tablet 66198d766e6e4ceb820bc92bcca67d85 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e7dba13b3c454a2ca5a93e5c5cb62935]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:25.763953 6950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb023d72aa8d4248a64d4723e18a3cf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:25.764523 6951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66198d766e6e4ceb820bc92bcca67d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:25.764775 6949 tablet_service.cc:1467] Processing CreateTablet for tablet 331bcde3e6bf43c081af836c8cdebdc7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e7dba13b3c454a2ca5a93e5c5cb62935]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:25.765092 6949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 331bcde3e6bf43c081af836c8cdebdc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:25.768795 6820 tablet_service.cc:1467] Processing CreateTablet for tablet 66198d766e6e4ceb820bc92bcca67d85 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e7dba13b3c454a2ca5a93e5c5cb62935]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:25.769089 6690 tablet_service.cc:1467] Processing CreateTablet for tablet 331bcde3e6bf43c081af836c8cdebdc7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e7dba13b3c454a2ca5a93e5c5cb62935]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:25.769245 6820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66198d766e6e4ceb820bc92bcca67d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:25.769240 6560 tablet_service.cc:1467] Processing CreateTablet for tablet 66198d766e6e4ceb820bc92bcca67d85 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e7dba13b3c454a2ca5a93e5c5cb62935]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:25.769449 6690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 331bcde3e6bf43c081af836c8cdebdc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:25.769568 6560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66198d766e6e4ceb820bc92bcca67d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:25.772769 6559 tablet_service.cc:1467] Processing CreateTablet for tablet 331bcde3e6bf43c081af836c8cdebdc7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e7dba13b3c454a2ca5a93e5c5cb62935]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:25.773101 6559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 331bcde3e6bf43c081af836c8cdebdc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:25.779757 7743 tablet_bootstrap.cc:492] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae: Bootstrap starting.
I20240502 21:27:25.779920 7744 tablet_bootstrap.cc:492] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2: Bootstrap starting.
I20240502 21:27:25.781255 7743 tablet_bootstrap.cc:654] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:25.782742 7744 tablet_bootstrap.cc:654] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:25.791841 7742 tablet_bootstrap.cc:492] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef: Bootstrap starting.
I20240502 21:27:25.792768 7744 tablet_bootstrap.cc:492] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2: No bootstrap required, opened a new log
I20240502 21:27:25.792901 7744 ts_tablet_manager.cc:1367] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20240502 21:27:25.793656 7744 raft_consensus.cc:348] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:25.793886 7744 raft_consensus.cc:374] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:25.793918 7743 tablet_bootstrap.cc:492] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae: No bootstrap required, opened a new log
I20240502 21:27:25.793969 7744 raft_consensus.cc:729] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Initialized, Role: FOLLOWER
I20240502 21:27:25.794044 7743 ts_tablet_manager.cc:1367] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.003s
I20240502 21:27:25.794203 7744 consensus_queue.cc:260] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:25.794391 7742 tablet_bootstrap.cc:654] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:25.794775 7744 ts_tablet_manager.cc:1398] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:25.794826 7743 raft_consensus.cc:348] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:25.794988 7743 raft_consensus.cc:374] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:25.795073 7743 raft_consensus.cc:729] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Initialized, Role: FOLLOWER
I20240502 21:27:25.795295 7743 consensus_queue.cc:260] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:25.803495 7742 tablet_bootstrap.cc:492] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef: No bootstrap required, opened a new log
I20240502 21:27:25.803761 7742 ts_tablet_manager.cc:1367] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20240502 21:27:25.805087 7742 raft_consensus.cc:348] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:25.805462 7742 raft_consensus.cc:374] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:25.805609 7742 raft_consensus.cc:729] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Initialized, Role: FOLLOWER
I20240502 21:27:25.805907 7742 consensus_queue.cc:260] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:25.808991 7744 tablet_bootstrap.cc:492] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2: Bootstrap starting.
I20240502 21:27:25.809588 7742 ts_tablet_manager.cc:1398] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 21:27:25.810529 7744 tablet_bootstrap.cc:654] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:25.811872 7742 tablet_bootstrap.cc:492] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef: Bootstrap starting.
I20240502 21:27:25.813678 7742 tablet_bootstrap.cc:654] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:25.814630 7745 tablet_bootstrap.cc:492] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720: Bootstrap starting.
I20240502 21:27:25.816864 7745 tablet_bootstrap.cc:654] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:25.818084 7743 ts_tablet_manager.cc:1398] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae: Time spent starting tablet: real 0.024s user 0.005s sys 0.018s
I20240502 21:27:25.818341 7743 tablet_bootstrap.cc:492] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae: Bootstrap starting.
I20240502 21:27:25.819748 7743 tablet_bootstrap.cc:654] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:25.822086 7742 tablet_bootstrap.cc:492] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef: No bootstrap required, opened a new log
I20240502 21:27:25.822258 7742 ts_tablet_manager.cc:1367] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240502 21:27:25.823071 7742 raft_consensus.cc:348] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:25.823305 7742 raft_consensus.cc:374] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:25.823397 7742 raft_consensus.cc:729] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Initialized, Role: FOLLOWER
I20240502 21:27:25.823803 7742 consensus_queue.cc:260] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:25.825289 7742 ts_tablet_manager.cc:1398] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:25.826272 7744 tablet_bootstrap.cc:492] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2: No bootstrap required, opened a new log
I20240502 21:27:25.826491 7744 ts_tablet_manager.cc:1367] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.004s
I20240502 21:27:25.827462 7744 raft_consensus.cc:348] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:25.827739 7744 raft_consensus.cc:374] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:25.827817 7744 raft_consensus.cc:729] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Initialized, Role: FOLLOWER
I20240502 21:27:25.828065 7744 consensus_queue.cc:260] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:25.828505 7743 tablet_bootstrap.cc:492] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae: No bootstrap required, opened a new log
I20240502 21:27:25.828652 7744 ts_tablet_manager.cc:1398] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:25.828671 7743 ts_tablet_manager.cc:1367] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20240502 21:27:25.829257 7743 raft_consensus.cc:348] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:25.829424 7744 tablet_bootstrap.cc:492] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2: Bootstrap starting.
I20240502 21:27:25.829448 7743 raft_consensus.cc:374] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:25.829525 7743 raft_consensus.cc:729] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Initialized, Role: FOLLOWER
I20240502 21:27:25.829819 7743 consensus_queue.cc:260] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:25.830318 7743 ts_tablet_manager.cc:1398] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:25.831552 7744 tablet_bootstrap.cc:654] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:25.835439 7743 tablet_bootstrap.cc:492] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae: Bootstrap starting.
I20240502 21:27:25.838356 7743 tablet_bootstrap.cc:654] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:25.837260 7745 tablet_bootstrap.cc:492] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720: No bootstrap required, opened a new log
I20240502 21:27:25.838702 7745 ts_tablet_manager.cc:1367] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.008s
I20240502 21:27:25.839437 7745 raft_consensus.cc:348] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:25.839645 7745 raft_consensus.cc:374] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:25.839722 7745 raft_consensus.cc:729] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Initialized, Role: FOLLOWER
I20240502 21:27:25.839917 7745 consensus_queue.cc:260] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:25.840540 7745 ts_tablet_manager.cc:1398] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:27:25.842156 7744 tablet_bootstrap.cc:492] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2: No bootstrap required, opened a new log
I20240502 21:27:25.842286 7744 ts_tablet_manager.cc:1367] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20240502 21:27:25.842948 7744 raft_consensus.cc:348] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:25.843097 7744 raft_consensus.cc:374] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:25.843178 7744 raft_consensus.cc:729] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Initialized, Role: FOLLOWER
I20240502 21:27:25.843367 7744 consensus_queue.cc:260] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:25.843856 7744 ts_tablet_manager.cc:1398] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:27:25.845856 7743 tablet_bootstrap.cc:492] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae: No bootstrap required, opened a new log
I20240502 21:27:25.845985 7743 ts_tablet_manager.cc:1367] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.005s
I20240502 21:27:25.846621 7743 raft_consensus.cc:348] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:25.846814 7743 raft_consensus.cc:374] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:25.846979 7743 raft_consensus.cc:729] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Initialized, Role: FOLLOWER
I20240502 21:27:25.847266 7743 consensus_queue.cc:260] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:25.847827 7743 ts_tablet_manager.cc:1398] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:25.866533 7749 raft_consensus.cc:482] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:25.866658 7749 raft_consensus.cc:504] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:25.867960 7749 leader_election.cc:288] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:25.884621 6971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331bcde3e6bf43c081af836c8cdebdc7" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a838c00b544bf68ec4cc9c513212b2" is_pre_election: true
I20240502 21:27:25.884842 6971 raft_consensus.cc:2454] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 0.
I20240502 21:27:25.884891 6710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331bcde3e6bf43c081af836c8cdebdc7" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" is_pre_election: true
I20240502 21:27:25.885114 6710 raft_consensus.cc:2454] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 0.
I20240502 21:27:25.885275 6514 leader_election.cc:302] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2; no voters:
I20240502 21:27:25.885661 7749 raft_consensus.cc:2781] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:25.885769 7749 raft_consensus.cc:482] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:25.885859 7749 raft_consensus.cc:3037] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:25.887822 7749 raft_consensus.cc:504] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:25.888352 7749 leader_election.cc:288] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 1 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:25.888679 6971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331bcde3e6bf43c081af836c8cdebdc7" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:25.888907 6971 raft_consensus.cc:3037] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:25.888927 6710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331bcde3e6bf43c081af836c8cdebdc7" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:25.889076 6710 raft_consensus.cc:3037] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:25.890910 6971 raft_consensus.cc:2454] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 1.
I20240502 21:27:25.890933 6710 raft_consensus.cc:2454] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 1.
I20240502 21:27:25.891256 6515 leader_election.cc:302] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:25.891530 7749 raft_consensus.cc:2781] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:25.891891 7749 raft_consensus.cc:686] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 1 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:25.892379 7749 consensus_queue.cc:237] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:25.895804 6437 catalog_manager.cc:5484] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 0 to 1, leader changed from <none> to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 1 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:26.020025 7602 raft_consensus.cc:482] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:26.020215 7602 raft_consensus.cc:504] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:26.020908 7602 leader_election.cc:288] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:26.021502 6710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb023d72aa8d4248a64d4723e18a3cf6" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" is_pre_election: true
I20240502 21:27:26.021788 6710 raft_consensus.cc:2454] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 0.
I20240502 21:27:26.022214 6906 leader_election.cc:302] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:26.022501 7602 raft_consensus.cc:2781] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:26.022639 7602 raft_consensus.cc:482] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:26.022773 7602 raft_consensus.cc:3037] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:26.025089 7602 raft_consensus.cc:504] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:26.025794 7602 leader_election.cc:288] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 1 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:26.026335 6710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb023d72aa8d4248a64d4723e18a3cf6" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:26.026597 6710 raft_consensus.cc:3037] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:26.028450 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb023d72aa8d4248a64d4723e18a3cf6" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bc1a0939fe49348a18d3da18f9feae" is_pre_election: true
I20240502 21:27:26.028705 6580 raft_consensus.cc:2454] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 0.
I20240502 21:27:26.028793 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb023d72aa8d4248a64d4723e18a3cf6" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:26.028919 6710 raft_consensus.cc:2454] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 1.
I20240502 21:27:26.029012 6579 raft_consensus.cc:3037] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:26.029588 6906 leader_election.cc:302] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:26.029850 7602 raft_consensus.cc:2781] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:26.030227 7602 raft_consensus.cc:686] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:26.030526 7602 consensus_queue.cc:237] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:26.031564 6579 raft_consensus.cc:2454] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 1.
I20240502 21:27:26.033275 6437 catalog_manager.cc:5484] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 1 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:26.213004 7602 raft_consensus.cc:482] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:26.213194 7602 raft_consensus.cc:504] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:26.213699 7602 leader_election.cc:288] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:26.214179 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66198d766e6e4ceb820bc92bcca67d85" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9793130a8e4f45388357b8d2e9f47720" is_pre_election: true
I20240502 21:27:26.214283 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66198d766e6e4ceb820bc92bcca67d85" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bc1a0939fe49348a18d3da18f9feae" is_pre_election: true
I20240502 21:27:26.214401 6841 raft_consensus.cc:2454] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 0.
I20240502 21:27:26.214538 6579 raft_consensus.cc:2454] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 0.
I20240502 21:27:26.214854 6904 leader_election.cc:302] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:26.215096 7602 raft_consensus.cc:2781] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:26.215188 7602 raft_consensus.cc:482] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:26.215288 7602 raft_consensus.cc:3037] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:26.217147 7602 raft_consensus.cc:504] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:26.217702 7602 leader_election.cc:288] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 1 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:26.218060 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66198d766e6e4ceb820bc92bcca67d85" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:26.218243 6841 raft_consensus.cc:3037] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:26.218187 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66198d766e6e4ceb820bc92bcca67d85" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:26.218380 6579 raft_consensus.cc:3037] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:26.219846 6841 raft_consensus.cc:2454] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 1.
I20240502 21:27:26.220028 6579 raft_consensus.cc:2454] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 1.
I20240502 21:27:26.220180 6904 leader_election.cc:302] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:26.220387 7602 raft_consensus.cc:2781] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:26.220558 7602 raft_consensus.cc:686] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:26.220789 7602 consensus_queue.cc:237] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:26.223452 6437 catalog_manager.cc:5484] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 1 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:26.316270 7749 consensus_queue.cc:1035] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:26.332242 7749 consensus_queue.cc:1035] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:26.546891 7767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:26.547161 7767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:26.553898 7767 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:27:26.557358 7602 consensus_queue.cc:1035] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:26.563642 7764 consensus_queue.cc:1035] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240502 21:27:26.649577 7764 consensus_queue.cc:1035] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:26.685832 7602 consensus_queue.cc:1035] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 268.808 ms
time per row: 0.134404 ms
I20240502 21:27:26.955633 6439 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48454:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:26.963313 6820 tablet_service.cc:1467] Processing CreateTablet for tablet 1c98cbfbd0a44335b75b228c217f9132 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3852001d56f643749bad82f8ef484b1d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:26.963608 6690 tablet_service.cc:1467] Processing CreateTablet for tablet 1c98cbfbd0a44335b75b228c217f9132 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3852001d56f643749bad82f8ef484b1d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:26.963773 6820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c98cbfbd0a44335b75b228c217f9132. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:26.964120 6690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c98cbfbd0a44335b75b228c217f9132. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:26.964620 6689 tablet_service.cc:1467] Processing CreateTablet for tablet a09d2491d9dc4cbfb20f7adc1087b375 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3852001d56f643749bad82f8ef484b1d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:26.964983 6689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a09d2491d9dc4cbfb20f7adc1087b375. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:26.966120 6819 tablet_service.cc:1467] Processing CreateTablet for tablet 093f8fd73ec34d8b9954c1716e910a78 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3852001d56f643749bad82f8ef484b1d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:26.966455 6819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 093f8fd73ec34d8b9954c1716e910a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:26.968845 6559 tablet_service.cc:1467] Processing CreateTablet for tablet 1c98cbfbd0a44335b75b228c217f9132 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3852001d56f643749bad82f8ef484b1d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:26.969311 6559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c98cbfbd0a44335b75b228c217f9132. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:26.970863 6949 tablet_service.cc:1467] Processing CreateTablet for tablet 093f8fd73ec34d8b9954c1716e910a78 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3852001d56f643749bad82f8ef484b1d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:26.971276 6949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 093f8fd73ec34d8b9954c1716e910a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:26.972856 6821 tablet_service.cc:1467] Processing CreateTablet for tablet a09d2491d9dc4cbfb20f7adc1087b375 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3852001d56f643749bad82f8ef484b1d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:26.973238 6821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a09d2491d9dc4cbfb20f7adc1087b375. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:26.976007 6950 tablet_service.cc:1467] Processing CreateTablet for tablet a09d2491d9dc4cbfb20f7adc1087b375 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3852001d56f643749bad82f8ef484b1d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:26.976471 6950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a09d2491d9dc4cbfb20f7adc1087b375. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:26.976532 6558 tablet_service.cc:1467] Processing CreateTablet for tablet 093f8fd73ec34d8b9954c1716e910a78 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3852001d56f643749bad82f8ef484b1d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:26.977036 6558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 093f8fd73ec34d8b9954c1716e910a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:26.984252 7832 tablet_bootstrap.cc:492] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae: Bootstrap starting.
I20240502 21:27:26.984802 7833 tablet_bootstrap.cc:492] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2: Bootstrap starting.
I20240502 21:27:26.986683 7833 tablet_bootstrap.cc:654] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:26.989162 7832 tablet_bootstrap.cc:654] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:26.993762 7831 tablet_bootstrap.cc:492] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720: Bootstrap starting.
I20240502 21:27:26.995065 7833 tablet_bootstrap.cc:492] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2: No bootstrap required, opened a new log
I20240502 21:27:26.995213 7833 ts_tablet_manager.cc:1367] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.007s
I20240502 21:27:26.995358 7836 tablet_bootstrap.cc:492] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef: Bootstrap starting.
I20240502 21:27:26.995559 7831 tablet_bootstrap.cc:654] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:26.995985 7833 raft_consensus.cc:348] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:26.996201 7833 raft_consensus.cc:374] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:26.996218 7832 tablet_bootstrap.cc:492] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae: No bootstrap required, opened a new log
I20240502 21:27:26.996284 7833 raft_consensus.cc:729] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Initialized, Role: FOLLOWER
I20240502 21:27:26.996341 7832 ts_tablet_manager.cc:1367] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20240502 21:27:26.996582 7833 consensus_queue.cc:260] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:26.997021 7836 tablet_bootstrap.cc:654] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:26.997229 7832 raft_consensus.cc:348] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:26.997426 7832 raft_consensus.cc:374] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:26.997519 7832 raft_consensus.cc:729] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Initialized, Role: FOLLOWER
I20240502 21:27:26.997795 7832 consensus_queue.cc:260] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:26.999167 7832 ts_tablet_manager.cc:1398] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:26.999447 7832 tablet_bootstrap.cc:492] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae: Bootstrap starting.
I20240502 21:27:27.001219 7832 tablet_bootstrap.cc:654] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:27.002033 7833 ts_tablet_manager.cc:1398] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 21:27:27.005544 7831 tablet_bootstrap.cc:492] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720: No bootstrap required, opened a new log
I20240502 21:27:27.005719 7831 ts_tablet_manager.cc:1367] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20240502 21:27:27.006668 7831 raft_consensus.cc:348] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:27.006996 7831 raft_consensus.cc:374] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:27.007150 7831 raft_consensus.cc:729] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Initialized, Role: FOLLOWER
I20240502 21:27:27.007372 7833 tablet_bootstrap.cc:492] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2: Bootstrap starting.
I20240502 21:27:27.007572 7831 consensus_queue.cc:260] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:27.008417 7831 ts_tablet_manager.cc:1398] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:27:27.008993 7833 tablet_bootstrap.cc:654] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:27.009972 7832 tablet_bootstrap.cc:492] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae: No bootstrap required, opened a new log
I20240502 21:27:27.010139 7832 ts_tablet_manager.cc:1367] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240502 21:27:27.010375 7831 tablet_bootstrap.cc:492] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720: Bootstrap starting.
I20240502 21:27:27.010756 7832 raft_consensus.cc:348] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:27.010943 7832 raft_consensus.cc:374] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:27.011008 7832 raft_consensus.cc:729] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Initialized, Role: FOLLOWER
I20240502 21:27:27.011245 7832 consensus_queue.cc:260] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:27.012203 7832 ts_tablet_manager.cc:1398] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:27.013278 7833 tablet_bootstrap.cc:492] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2: No bootstrap required, opened a new log
I20240502 21:27:27.013417 7833 ts_tablet_manager.cc:1367] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 21:27:27.013727 7831 tablet_bootstrap.cc:654] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:27.014104 7833 raft_consensus.cc:348] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:27.014298 7833 raft_consensus.cc:374] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:27.014369 7833 raft_consensus.cc:729] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Initialized, Role: FOLLOWER
I20240502 21:27:27.014639 7833 consensus_queue.cc:260] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:27.015197 7833 ts_tablet_manager.cc:1398] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:27.015311 7836 tablet_bootstrap.cc:492] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef: No bootstrap required, opened a new log
I20240502 21:27:27.015445 7836 ts_tablet_manager.cc:1367] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.008s
I20240502 21:27:27.015755 7831 tablet_bootstrap.cc:492] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720: No bootstrap required, opened a new log
I20240502 21:27:27.015895 7831 ts_tablet_manager.cc:1367] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240502 21:27:27.016088 7836 raft_consensus.cc:348] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:27.016227 7836 raft_consensus.cc:374] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:27.016319 7836 raft_consensus.cc:729] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Initialized, Role: FOLLOWER
I20240502 21:27:27.016649 7831 raft_consensus.cc:348] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:27.016685 7836 consensus_queue.cc:260] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:27.016862 7831 raft_consensus.cc:374] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:27.016949 7831 raft_consensus.cc:729] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Initialized, Role: FOLLOWER
I20240502 21:27:27.017189 7831 consensus_queue.cc:260] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:27.017576 7836 ts_tablet_manager.cc:1398] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:27.017706 7831 ts_tablet_manager.cc:1398] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:27.017843 7836 tablet_bootstrap.cc:492] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef: Bootstrap starting.
I20240502 21:27:27.017963 7831 tablet_bootstrap.cc:492] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720: Bootstrap starting.
I20240502 21:27:27.019704 7831 tablet_bootstrap.cc:654] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:27.022090 7831 tablet_bootstrap.cc:492] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720: No bootstrap required, opened a new log
I20240502 21:27:27.022352 7831 ts_tablet_manager.cc:1367] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:27:27.022982 7831 raft_consensus.cc:348] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:27.023170 7836 tablet_bootstrap.cc:654] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:27.023176 7831 raft_consensus.cc:374] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:27.023273 7831 raft_consensus.cc:729] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Initialized, Role: FOLLOWER
I20240502 21:27:27.023502 7831 consensus_queue.cc:260] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:27.024586 7831 ts_tablet_manager.cc:1398] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:27.025055 7836 tablet_bootstrap.cc:492] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef: No bootstrap required, opened a new log
I20240502 21:27:27.025192 7836 ts_tablet_manager.cc:1367] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 21:27:27.025744 7836 raft_consensus.cc:348] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:27.025890 7836 raft_consensus.cc:374] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:27.025952 7836 raft_consensus.cc:729] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Initialized, Role: FOLLOWER
I20240502 21:27:27.026175 7836 consensus_queue.cc:260] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:27.026558 7836 ts_tablet_manager.cc:1398] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:27:27.029436 7771 raft_consensus.cc:482] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:27.029579 7771 raft_consensus.cc:504] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:27.030382 7771 leader_election.cc:288] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:27.031390 6971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093f8fd73ec34d8b9954c1716e910a78" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a838c00b544bf68ec4cc9c513212b2" is_pre_election: true
I20240502 21:27:27.031574 6971 raft_consensus.cc:2454] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 0.
I20240502 21:27:27.031971 6514 leader_election.cc:302] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2; no voters:
I20240502 21:27:27.032260 7771 raft_consensus.cc:2781] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:27.032392 7771 raft_consensus.cc:482] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:27.032506 7771 raft_consensus.cc:3037] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:27.034468 7771 raft_consensus.cc:504] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:27.035192 7771 leader_election.cc:288] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 1 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:27.035610 6971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093f8fd73ec34d8b9954c1716e910a78" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:27.035807 6971 raft_consensus.cc:3037] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:27.037271 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093f8fd73ec34d8b9954c1716e910a78" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9793130a8e4f45388357b8d2e9f47720" is_pre_election: true
I20240502 21:27:27.037493 6841 raft_consensus.cc:2454] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 0.
I20240502 21:27:27.037427 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093f8fd73ec34d8b9954c1716e910a78" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:27.037722 6840 raft_consensus.cc:3037] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:27.038166 6971 raft_consensus.cc:2454] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 1.
I20240502 21:27:27.039664 6840 raft_consensus.cc:2454] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 1.
I20240502 21:27:27.039965 6514 leader_election.cc:302] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2; no voters:
I20240502 21:27:27.040263 7771 raft_consensus.cc:2781] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:27.040413 7771 raft_consensus.cc:686] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 1 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:27.040685 7771 consensus_queue.cc:237] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:27.043504 6439 catalog_manager.cc:5484] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 0 to 1, leader changed from <none> to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 1 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:27.112663 7726 raft_consensus.cc:482] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:27.112805 7726 raft_consensus.cc:504] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:27.113369 7726 leader_election.cc:288] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:27.113793 6971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09d2491d9dc4cbfb20f7adc1087b375" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a838c00b544bf68ec4cc9c513212b2" is_pre_election: true
I20240502 21:27:27.113816 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09d2491d9dc4cbfb20f7adc1087b375" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" is_pre_election: true
I20240502 21:27:27.114038 6711 raft_consensus.cc:2454] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 0.
I20240502 21:27:27.114058 6971 raft_consensus.cc:2454] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 0.
I20240502 21:27:27.114482 6776 leader_election.cc:302] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:27.114755 7726 raft_consensus.cc:2781] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:27.114881 7726 raft_consensus.cc:482] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:27.114967 7726 raft_consensus.cc:3037] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:27.116879 7726 raft_consensus.cc:504] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:27.117372 7726 leader_election.cc:288] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:27.117758 6971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09d2491d9dc4cbfb20f7adc1087b375" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:27.117812 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09d2491d9dc4cbfb20f7adc1087b375" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:27.117950 6971 raft_consensus.cc:3037] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:27.117985 6711 raft_consensus.cc:3037] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:27.119689 6971 raft_consensus.cc:2454] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 1.
I20240502 21:27:27.119738 6711 raft_consensus.cc:2454] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 1.
I20240502 21:27:27.120059 6775 leader_election.cc:302] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:27.120332 7726 raft_consensus.cc:2781] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:27.120496 7726 raft_consensus.cc:686] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 1 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:27.120746 7726 consensus_queue.cc:237] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:27.123091 6439 catalog_manager.cc:5484] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 1 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:27.343991 7771 raft_consensus.cc:482] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:27.344172 7771 raft_consensus.cc:504] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:27.344763 7771 leader_election.cc:288] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:27.345172 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c98cbfbd0a44335b75b228c217f9132" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9793130a8e4f45388357b8d2e9f47720" is_pre_election: true
I20240502 21:27:27.345259 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c98cbfbd0a44335b75b228c217f9132" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" is_pre_election: true
I20240502 21:27:27.345381 6840 raft_consensus.cc:2454] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 0.
I20240502 21:27:27.345469 6711 raft_consensus.cc:2454] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 0.
I20240502 21:27:27.345789 6513 leader_election.cc:302] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:27.346035 7771 raft_consensus.cc:2781] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:27.346112 7771 raft_consensus.cc:482] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:27.346179 7771 raft_consensus.cc:3037] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:27.348127 7771 raft_consensus.cc:504] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:27.348639 7771 leader_election.cc:288] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 1 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:27.348968 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c98cbfbd0a44335b75b228c217f9132" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:27.349076 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c98cbfbd0a44335b75b228c217f9132" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:27.349138 6840 raft_consensus.cc:3037] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:27.349254 6711 raft_consensus.cc:3037] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:27.350766 6840 raft_consensus.cc:2454] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 1.
I20240502 21:27:27.350955 6711 raft_consensus.cc:2454] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 1.
I20240502 21:27:27.351095 6513 leader_election.cc:302] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:27.351366 7771 raft_consensus.cc:2781] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:27.351542 7771 raft_consensus.cc:686] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 1 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:27.351768 7771 consensus_queue.cc:237] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:27.354018 6439 catalog_manager.cc:5484] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 0 to 1, leader changed from <none> to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 1 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:27.506842 7771 consensus_queue.cc:1035] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:27.511652 7771 consensus_queue.cc:1035] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:27.522219 7726 consensus_queue.cc:1035] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:27.526854 7842 consensus_queue.cc:1035] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:27.750524 7850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:27.750768 7850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 21:27:27.756026 7771 consensus_queue.cc:1035] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:27.758908 7850 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:27:27.760726 7760 consensus_queue.cc:1035] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 236.432 ms
time per row: 0.118216 ms
I20240502 21:27:28.129536 6434 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48454:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:28.137465 6558 tablet_service.cc:1467] Processing CreateTablet for tablet 169899da82594d9cb673f8db74764ca2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6dfc103c32684c78adb67657608ec9bd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:28.137642 6690 tablet_service.cc:1467] Processing CreateTablet for tablet 169899da82594d9cb673f8db74764ca2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6dfc103c32684c78adb67657608ec9bd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:28.137977 6558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 169899da82594d9cb673f8db74764ca2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:28.138053 6820 tablet_service.cc:1467] Processing CreateTablet for tablet 169899da82594d9cb673f8db74764ca2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6dfc103c32684c78adb67657608ec9bd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:28.138139 6690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 169899da82594d9cb673f8db74764ca2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:28.138432 6820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 169899da82594d9cb673f8db74764ca2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:28.140302 6689 tablet_service.cc:1467] Processing CreateTablet for tablet 875f83053fee45b3bc1921b24745e9da (DEFAULT_TABLE table=rebalance_test_table_4 [id=6dfc103c32684c78adb67657608ec9bd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:28.140635 6689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 875f83053fee45b3bc1921b24745e9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:28.140789 6821 tablet_service.cc:1467] Processing CreateTablet for tablet 980f9cc0cdc6452d8009e9f75a0d4275 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6dfc103c32684c78adb67657608ec9bd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:28.141113 6821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 980f9cc0cdc6452d8009e9f75a0d4275. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:28.141135 6691 tablet_service.cc:1467] Processing CreateTablet for tablet 980f9cc0cdc6452d8009e9f75a0d4275 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6dfc103c32684c78adb67657608ec9bd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:28.141441 6691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 980f9cc0cdc6452d8009e9f75a0d4275. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:28.144973 6949 tablet_service.cc:1467] Processing CreateTablet for tablet 875f83053fee45b3bc1921b24745e9da (DEFAULT_TABLE table=rebalance_test_table_4 [id=6dfc103c32684c78adb67657608ec9bd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:28.145483 6949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 875f83053fee45b3bc1921b24745e9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:28.148896 6950 tablet_service.cc:1467] Processing CreateTablet for tablet 980f9cc0cdc6452d8009e9f75a0d4275 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6dfc103c32684c78adb67657608ec9bd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:28.149309 6950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 980f9cc0cdc6452d8009e9f75a0d4275. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:28.151569 7918 tablet_bootstrap.cc:492] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef: Bootstrap starting.
I20240502 21:27:28.152402 6559 tablet_service.cc:1467] Processing CreateTablet for tablet 875f83053fee45b3bc1921b24745e9da (DEFAULT_TABLE table=rebalance_test_table_4 [id=6dfc103c32684c78adb67657608ec9bd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:28.152809 6559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 875f83053fee45b3bc1921b24745e9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:28.153474 7918 tablet_bootstrap.cc:654] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:28.165833 7917 tablet_bootstrap.cc:492] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720: Bootstrap starting.
I20240502 21:27:28.167501 7921 tablet_bootstrap.cc:492] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae: Bootstrap starting.
I20240502 21:27:28.168103 7917 tablet_bootstrap.cc:654] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:28.168845 7918 tablet_bootstrap.cc:492] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef: No bootstrap required, opened a new log
I20240502 21:27:28.169111 7918 ts_tablet_manager.cc:1367] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.005s
I20240502 21:27:28.169435 7920 tablet_bootstrap.cc:492] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2: Bootstrap starting.
I20240502 21:27:28.170379 7921 tablet_bootstrap.cc:654] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:28.170426 7918 raft_consensus.cc:348] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:28.170785 7918 raft_consensus.cc:374] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:28.170881 7918 raft_consensus.cc:729] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Initialized, Role: FOLLOWER
I20240502 21:27:28.171082 7918 consensus_queue.cc:260] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:28.171578 7917 tablet_bootstrap.cc:492] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720: No bootstrap required, opened a new log
I20240502 21:27:28.171728 7917 ts_tablet_manager.cc:1367] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20240502 21:27:28.172097 7920 tablet_bootstrap.cc:654] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:28.172508 7917 raft_consensus.cc:348] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:28.172713 7917 raft_consensus.cc:374] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:28.172821 7917 raft_consensus.cc:729] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Initialized, Role: FOLLOWER
I20240502 21:27:28.173110 7917 consensus_queue.cc:260] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:28.173801 7917 ts_tablet_manager.cc:1398] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:28.174093 7917 tablet_bootstrap.cc:492] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720: Bootstrap starting.
I20240502 21:27:28.174394 7918 ts_tablet_manager.cc:1398] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:27:28.175390 7917 tablet_bootstrap.cc:654] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:28.177636 7918 tablet_bootstrap.cc:492] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef: Bootstrap starting.
I20240502 21:27:28.179576 7918 tablet_bootstrap.cc:654] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:28.179929 7921 tablet_bootstrap.cc:492] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae: No bootstrap required, opened a new log
I20240502 21:27:28.180176 7921 ts_tablet_manager.cc:1367] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20240502 21:27:28.181206 7921 raft_consensus.cc:348] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:28.181509 7921 raft_consensus.cc:374] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:28.181654 7921 raft_consensus.cc:729] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Initialized, Role: FOLLOWER
I20240502 21:27:28.181991 7921 consensus_queue.cc:260] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:28.182866 7921 ts_tablet_manager.cc:1398] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:27:28.183264 7921 tablet_bootstrap.cc:492] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae: Bootstrap starting.
I20240502 21:27:28.185288 7921 tablet_bootstrap.cc:654] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:28.186452 7920 tablet_bootstrap.cc:492] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2: No bootstrap required, opened a new log
I20240502 21:27:28.186995 7917 tablet_bootstrap.cc:492] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720: No bootstrap required, opened a new log
I20240502 21:27:28.187027 7920 ts_tablet_manager.cc:1367] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.004s
I20240502 21:27:28.187220 7918 tablet_bootstrap.cc:492] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef: No bootstrap required, opened a new log
I20240502 21:27:28.187235 7917 ts_tablet_manager.cc:1367] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.002s
I20240502 21:27:28.187443 7918 ts_tablet_manager.cc:1367] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20240502 21:27:28.188004 7920 raft_consensus.cc:348] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:28.188247 7920 raft_consensus.cc:374] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:28.188222 7917 raft_consensus.cc:348] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:28.188246 7918 raft_consensus.cc:348] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:28.188367 7920 raft_consensus.cc:729] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Initialized, Role: FOLLOWER
I20240502 21:27:28.188473 7917 raft_consensus.cc:374] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:28.188498 7918 raft_consensus.cc:374] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:28.188557 7917 raft_consensus.cc:729] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Initialized, Role: FOLLOWER
I20240502 21:27:28.188570 7918 raft_consensus.cc:729] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Initialized, Role: FOLLOWER
I20240502 21:27:28.188625 7920 consensus_queue.cc:260] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:28.188810 7917 consensus_queue.cc:260] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:28.188859 7918 consensus_queue.cc:260] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:28.189412 7917 ts_tablet_manager.cc:1398] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:28.189970 7920 ts_tablet_manager.cc:1398] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:28.190299 7920 tablet_bootstrap.cc:492] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2: Bootstrap starting.
I20240502 21:27:28.190662 7918 ts_tablet_manager.cc:1398] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:27:28.191037 7918 tablet_bootstrap.cc:492] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef: Bootstrap starting.
I20240502 21:27:28.191936 7920 tablet_bootstrap.cc:654] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:28.194059 7918 tablet_bootstrap.cc:654] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:28.198899 7918 tablet_bootstrap.cc:492] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef: No bootstrap required, opened a new log
I20240502 21:27:28.199074 7918 ts_tablet_manager.cc:1367] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20240502 21:27:28.199999 7918 raft_consensus.cc:348] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:28.200297 7918 raft_consensus.cc:374] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:28.200481 7918 raft_consensus.cc:729] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Initialized, Role: FOLLOWER
I20240502 21:27:28.200798 7918 consensus_queue.cc:260] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:28.201483 7918 ts_tablet_manager.cc:1398] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:28.202809 7921 tablet_bootstrap.cc:492] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae: No bootstrap required, opened a new log
I20240502 21:27:28.203430 7921 ts_tablet_manager.cc:1367] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.007s
I20240502 21:27:28.203850 7920 tablet_bootstrap.cc:492] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2: No bootstrap required, opened a new log
I20240502 21:27:28.203984 7920 ts_tablet_manager.cc:1367] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20240502 21:27:28.204046 7921 raft_consensus.cc:348] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:28.204252 7921 raft_consensus.cc:374] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:28.204370 7921 raft_consensus.cc:729] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Initialized, Role: FOLLOWER
I20240502 21:27:28.204643 7921 consensus_queue.cc:260] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:28.204634 7920 raft_consensus.cc:348] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:28.204828 7920 raft_consensus.cc:374] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:28.205098 7921 ts_tablet_manager.cc:1398] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:27:28.205509 7892 raft_consensus.cc:482] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:28.205569 7920 raft_consensus.cc:729] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Initialized, Role: FOLLOWER
I20240502 21:27:28.205646 7892 raft_consensus.cc:504] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:28.205807 7920 consensus_queue.cc:260] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:28.206290 7920 ts_tablet_manager.cc:1398] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:28.206271 7892 leader_election.cc:288] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:28.206640 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875f83053fee45b3bc1921b24745e9da" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a838c00b544bf68ec4cc9c513212b2" is_pre_election: true
I20240502 21:27:28.206842 6970 raft_consensus.cc:2454] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 0.
I20240502 21:27:28.207195 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875f83053fee45b3bc1921b24745e9da" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" is_pre_election: true
I20240502 21:27:28.207398 6711 raft_consensus.cc:2454] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 0.
I20240502 21:27:28.207824 6514 leader_election.cc:302] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2; no voters:
I20240502 21:27:28.208112 7892 raft_consensus.cc:2781] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:28.208221 7892 raft_consensus.cc:482] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:28.208297 7892 raft_consensus.cc:3037] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:28.210561 7892 raft_consensus.cc:504] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:28.210944 7892 leader_election.cc:288] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 1 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:28.211390 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875f83053fee45b3bc1921b24745e9da" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:28.211519 6970 raft_consensus.cc:3037] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:28.212574 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875f83053fee45b3bc1921b24745e9da" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:28.212730 6711 raft_consensus.cc:3037] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:28.213184 6970 raft_consensus.cc:2454] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 1.
I20240502 21:27:28.213493 6514 leader_election.cc:302] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2; no voters:
I20240502 21:27:28.213714 7892 raft_consensus.cc:2781] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:28.213841 7892 raft_consensus.cc:686] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 1 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:28.214097 7892 consensus_queue.cc:237] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:28.214794 6711 raft_consensus.cc:2454] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 1.
I20240502 21:27:28.216620 6439 catalog_manager.cc:5484] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 0 to 1, leader changed from <none> to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 1 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:28.252511 7726 raft_consensus.cc:482] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:28.252635 7726 raft_consensus.cc:504] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:28.253208 7726 leader_election.cc:288] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:28.253499 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "980f9cc0cdc6452d8009e9f75a0d4275" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a838c00b544bf68ec4cc9c513212b2" is_pre_election: true
I20240502 21:27:28.253597 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "980f9cc0cdc6452d8009e9f75a0d4275" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" is_pre_election: true
I20240502 21:27:28.253686 6970 raft_consensus.cc:2454] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 0.
I20240502 21:27:28.253770 6711 raft_consensus.cc:2454] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 0.
I20240502 21:27:28.254011 6775 leader_election.cc:302] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:28.254249 7726 raft_consensus.cc:2781] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:28.254340 7726 raft_consensus.cc:482] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:28.254416 7726 raft_consensus.cc:3037] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:28.256032 7726 raft_consensus.cc:504] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:28.256510 7726 leader_election.cc:288] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:28.256868 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "980f9cc0cdc6452d8009e9f75a0d4275" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:28.256958 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "980f9cc0cdc6452d8009e9f75a0d4275" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:28.257074 6970 raft_consensus.cc:3037] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:28.257139 6711 raft_consensus.cc:3037] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:28.259191 6970 raft_consensus.cc:2454] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 1.
I20240502 21:27:28.259289 6711 raft_consensus.cc:2454] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 1.
I20240502 21:27:28.259505 6775 leader_election.cc:302] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:28.259755 7726 raft_consensus.cc:2781] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:28.259869 7726 raft_consensus.cc:686] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 1 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:28.260125 7726 consensus_queue.cc:237] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:28.262475 6439 catalog_manager.cc:5484] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 1 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:28.474943 7726 raft_consensus.cc:482] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:28.475117 7726 raft_consensus.cc:504] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:28.475617 7726 leader_election.cc:288] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:28.476071 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169899da82594d9cb673f8db74764ca2" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bc1a0939fe49348a18d3da18f9feae" is_pre_election: true
I20240502 21:27:28.476141 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169899da82594d9cb673f8db74764ca2" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" is_pre_election: true
I20240502 21:27:28.476269 6579 raft_consensus.cc:2454] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 0.
I20240502 21:27:28.476399 6711 raft_consensus.cc:2454] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 0.
I20240502 21:27:28.476742 6774 leader_election.cc:302] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:28.477036 7909 raft_consensus.cc:2781] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:28.477160 7909 raft_consensus.cc:482] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:28.477264 7909 raft_consensus.cc:3037] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:28.479282 7909 raft_consensus.cc:504] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:28.479769 7909 leader_election.cc:288] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:28.480122 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169899da82594d9cb673f8db74764ca2" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:28.480141 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169899da82594d9cb673f8db74764ca2" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:28.480270 6579 raft_consensus.cc:3037] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:28.480307 6711 raft_consensus.cc:3037] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:28.482141 6711 raft_consensus.cc:2454] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 1.
I20240502 21:27:28.482441 6579 raft_consensus.cc:2454] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 1.
I20240502 21:27:28.482599 6776 leader_election.cc:302] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:28.482844 7909 raft_consensus.cc:2781] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:28.482996 7909 raft_consensus.cc:686] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 1 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:28.483244 7909 consensus_queue.cc:237] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:28.485461 6439 catalog_manager.cc:5484] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 1 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:28.653753 7909 consensus_queue.cc:1035] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:28.663600 7726 consensus_queue.cc:1035] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:28.687901 7760 consensus_queue.cc:1035] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:28.698792 7892 consensus_queue.cc:1035] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:28.844010 7933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:28.844301 7933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:28.850833 7933 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 21:27:28.949499 7726 consensus_queue.cc:1035] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:28.974282 7937 consensus_queue.cc:1035] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:29.064999 6904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.226.131:33929, user_credentials={real_user=slave}} blocked reactor thread for 84047.1us
INSERT report
rows total: 2000
time total: 201.89 ms
time per row: 0.100945 ms
I20240502 21:27:29.198611 6431 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48454:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:29.206324 6559 tablet_service.cc:1467] Processing CreateTablet for tablet fbec2b1b80644a93b9239600157ad957 (DEFAULT_TABLE table=rebalance_test_table_5 [id=72f3ef3e2a8c47fba65b0c922c38bf44]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:29.206492 6950 tablet_service.cc:1467] Processing CreateTablet for tablet fbec2b1b80644a93b9239600157ad957 (DEFAULT_TABLE table=rebalance_test_table_5 [id=72f3ef3e2a8c47fba65b0c922c38bf44]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:29.206820 6559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbec2b1b80644a93b9239600157ad957. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:29.206944 6950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbec2b1b80644a93b9239600157ad957. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:29.211241 7999 tablet_bootstrap.cc:492] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2: Bootstrap starting.
I20240502 21:27:29.211280 6949 tablet_service.cc:1467] Processing CreateTablet for tablet 5b2e443bad344ce89ce69871406c5a29 (DEFAULT_TABLE table=rebalance_test_table_5 [id=72f3ef3e2a8c47fba65b0c922c38bf44]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:29.211596 6949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b2e443bad344ce89ce69871406c5a29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:29.211854 6820 tablet_service.cc:1467] Processing CreateTablet for tablet fbec2b1b80644a93b9239600157ad957 (DEFAULT_TABLE table=rebalance_test_table_5 [id=72f3ef3e2a8c47fba65b0c922c38bf44]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:29.212307 6820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbec2b1b80644a93b9239600157ad957. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:29.212895 7999 tablet_bootstrap.cc:654] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:29.212831 6821 tablet_service.cc:1467] Processing CreateTablet for tablet 5b2e443bad344ce89ce69871406c5a29 (DEFAULT_TABLE table=rebalance_test_table_5 [id=72f3ef3e2a8c47fba65b0c922c38bf44]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:29.213295 6821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b2e443bad344ce89ce69871406c5a29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:29.216846 6819 tablet_service.cc:1467] Processing CreateTablet for tablet 8fac59b25c164e658fe71cfdf7ca78cb (DEFAULT_TABLE table=rebalance_test_table_5 [id=72f3ef3e2a8c47fba65b0c922c38bf44]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:29.217270 6819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fac59b25c164e658fe71cfdf7ca78cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:29.217268 6689 tablet_service.cc:1467] Processing CreateTablet for tablet 8fac59b25c164e658fe71cfdf7ca78cb (DEFAULT_TABLE table=rebalance_test_table_5 [id=72f3ef3e2a8c47fba65b0c922c38bf44]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:29.217813 6689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fac59b25c164e658fe71cfdf7ca78cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:29.221635 6558 tablet_service.cc:1467] Processing CreateTablet for tablet 5b2e443bad344ce89ce69871406c5a29 (DEFAULT_TABLE table=rebalance_test_table_5 [id=72f3ef3e2a8c47fba65b0c922c38bf44]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:29.222155 6558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b2e443bad344ce89ce69871406c5a29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:29.224833 6560 tablet_service.cc:1467] Processing CreateTablet for tablet 8fac59b25c164e658fe71cfdf7ca78cb (DEFAULT_TABLE table=rebalance_test_table_5 [id=72f3ef3e2a8c47fba65b0c922c38bf44]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:29.225215 6560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fac59b25c164e658fe71cfdf7ca78cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:29.238780 8000 tablet_bootstrap.cc:492] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae: Bootstrap starting.
I20240502 21:27:29.240550 8000 tablet_bootstrap.cc:654] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:29.243841 8003 tablet_bootstrap.cc:492] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef: Bootstrap starting.
I20240502 21:27:29.246085 8003 tablet_bootstrap.cc:654] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:29.247733 8002 tablet_bootstrap.cc:492] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720: Bootstrap starting.
I20240502 21:27:29.249439 8002 tablet_bootstrap.cc:654] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:29.251626 8002 tablet_bootstrap.cc:492] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720: No bootstrap required, opened a new log
I20240502 21:27:29.251762 8002 ts_tablet_manager.cc:1367] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:27:29.252600 7999 tablet_bootstrap.cc:492] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2: No bootstrap required, opened a new log
I20240502 21:27:29.252669 8002 raft_consensus.cc:348] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:29.252854 7999 ts_tablet_manager.cc:1367] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2: Time spent bootstrapping tablet: real 0.042s user 0.014s sys 0.009s
I20240502 21:27:29.252924 8002 raft_consensus.cc:374] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:29.253046 8002 raft_consensus.cc:729] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Initialized, Role: FOLLOWER
I20240502 21:27:29.253280 8002 consensus_queue.cc:260] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:29.253863 8002 ts_tablet_manager.cc:1398] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:29.253818 7999 raft_consensus.cc:348] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:29.254093 7999 raft_consensus.cc:374] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:29.254215 7999 raft_consensus.cc:729] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Initialized, Role: FOLLOWER
I20240502 21:27:29.254516 7999 consensus_queue.cc:260] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:29.254739 8002 tablet_bootstrap.cc:492] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720: Bootstrap starting.
I20240502 21:27:29.255219 7999 ts_tablet_manager.cc:1398] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:29.255479 7999 tablet_bootstrap.cc:492] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2: Bootstrap starting.
I20240502 21:27:29.256968 7999 tablet_bootstrap.cc:654] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:29.257274 8000 tablet_bootstrap.cc:492] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae: No bootstrap required, opened a new log
I20240502 21:27:29.257422 8000 ts_tablet_manager.cc:1367] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.004s
I20240502 21:27:29.258183 8000 raft_consensus.cc:348] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:29.258395 8000 raft_consensus.cc:374] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:29.258482 8000 raft_consensus.cc:729] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Initialized, Role: FOLLOWER
I20240502 21:27:29.258705 8000 consensus_queue.cc:260] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:29.259245 8000 ts_tablet_manager.cc:1398] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:29.259816 8003 tablet_bootstrap.cc:492] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef: No bootstrap required, opened a new log
I20240502 21:27:29.260079 8003 ts_tablet_manager.cc:1367] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240502 21:27:29.261176 8003 raft_consensus.cc:348] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:29.261513 8003 raft_consensus.cc:374] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:29.261687 8003 raft_consensus.cc:729] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Initialized, Role: FOLLOWER
I20240502 21:27:29.261982 8003 consensus_queue.cc:260] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:29.262163 7999 tablet_bootstrap.cc:492] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2: No bootstrap required, opened a new log
I20240502 21:27:29.262312 7999 ts_tablet_manager.cc:1367] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 21:27:29.262714 8003 ts_tablet_manager.cc:1398] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:29.263291 8000 tablet_bootstrap.cc:492] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae: Bootstrap starting.
I20240502 21:27:29.263762 7999 raft_consensus.cc:348] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:29.264025 7999 raft_consensus.cc:374] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:29.264149 7999 raft_consensus.cc:729] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Initialized, Role: FOLLOWER
I20240502 21:27:29.264443 7999 consensus_queue.cc:260] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:29.264762 8000 tablet_bootstrap.cc:654] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:29.265071 7999 ts_tablet_manager.cc:1398] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:27:29.265830 8002 tablet_bootstrap.cc:654] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:29.267625 8002 tablet_bootstrap.cc:492] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720: No bootstrap required, opened a new log
I20240502 21:27:29.267823 8002 ts_tablet_manager.cc:1367] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240502 21:27:29.268560 8002 raft_consensus.cc:348] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:29.268777 8002 raft_consensus.cc:374] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:29.268908 8002 raft_consensus.cc:729] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Initialized, Role: FOLLOWER
I20240502 21:27:29.269405 8002 consensus_queue.cc:260] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:29.269903 8002 ts_tablet_manager.cc:1398] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:29.270170 8002 tablet_bootstrap.cc:492] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720: Bootstrap starting.
I20240502 21:27:29.270257 8000 tablet_bootstrap.cc:492] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae: No bootstrap required, opened a new log
I20240502 21:27:29.270429 8000 ts_tablet_manager.cc:1367] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 21:27:29.271106 8000 raft_consensus.cc:348] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:29.271327 8000 raft_consensus.cc:374] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:29.271404 8000 raft_consensus.cc:729] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Initialized, Role: FOLLOWER
I20240502 21:27:29.271606 8000 consensus_queue.cc:260] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:29.271756 8002 tablet_bootstrap.cc:654] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:29.274187 8000 ts_tablet_manager.cc:1398] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:29.274488 8000 tablet_bootstrap.cc:492] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae: Bootstrap starting.
I20240502 21:27:29.275760 8000 tablet_bootstrap.cc:654] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:29.278865 7942 raft_consensus.cc:482] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:29.279018 7942 raft_consensus.cc:504] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:29.279806 8002 tablet_bootstrap.cc:492] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720: No bootstrap required, opened a new log
I20240502 21:27:29.279788 7942 leader_election.cc:288] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:29.279989 8002 ts_tablet_manager.cc:1367] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240502 21:27:29.280189 8000 tablet_bootstrap.cc:492] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae: No bootstrap required, opened a new log
I20240502 21:27:29.280203 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fac59b25c164e658fe71cfdf7ca78cb" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9793130a8e4f45388357b8d2e9f47720" is_pre_election: true
I20240502 21:27:29.280315 8000 ts_tablet_manager.cc:1367] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20240502 21:27:29.280356 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fac59b25c164e658fe71cfdf7ca78cb" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" is_pre_election: true
I20240502 21:27:29.280560 6709 raft_consensus.cc:2454] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 0.
I20240502 21:27:29.280908 8000 raft_consensus.cc:348] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:29.280915 8002 raft_consensus.cc:348] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:29.281163 8002 raft_consensus.cc:374] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:29.281169 8000 raft_consensus.cc:374] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:29.281217 6515 leader_election.cc:302] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:29.281277 8000 raft_consensus.cc:729] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Initialized, Role: FOLLOWER
I20240502 21:27:29.281277 8002 raft_consensus.cc:729] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Initialized, Role: FOLLOWER
I20240502 21:27:29.281579 7942 raft_consensus.cc:2781] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:29.281483 8000 consensus_queue.cc:260] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:29.281719 7942 raft_consensus.cc:482] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:29.281636 8002 consensus_queue.cc:260] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:29.281805 7942 raft_consensus.cc:3037] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:29.282146 8002 ts_tablet_manager.cc:1398] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:29.282245 8000 ts_tablet_manager.cc:1398] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20240502 21:27:29.308717 6513 leader_election.cc:341] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:29.309329 7892 raft_consensus.cc:482] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:29.309492 7892 raft_consensus.cc:504] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:29.310166 7892 leader_election.cc:288] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:29.310561 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec2b1b80644a93b9239600157ad957" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a838c00b544bf68ec4cc9c513212b2" is_pre_election: true
I20240502 21:27:29.310776 6970 raft_consensus.cc:2454] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 0.
I20240502 21:27:29.310777 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec2b1b80644a93b9239600157ad957" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9793130a8e4f45388357b8d2e9f47720" is_pre_election: true
I20240502 21:27:29.310994 6840 raft_consensus.cc:2454] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 0.
I20240502 21:27:29.311025 7942 raft_consensus.cc:504] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:29.311236 6514 leader_election.cc:302] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2; no voters:
I20240502 21:27:29.311530 7892 raft_consensus.cc:2781] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:29.311745 7942 leader_election.cc:288] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 1 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:29.311746 7892 raft_consensus.cc:482] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:29.312211 7892 raft_consensus.cc:3037] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:29.312237 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fac59b25c164e658fe71cfdf7ca78cb" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:29.312263 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fac59b25c164e658fe71cfdf7ca78cb" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:29.312441 6840 raft_consensus.cc:3037] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:29.312580 6709 raft_consensus.cc:3037] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:29.314693 7892 raft_consensus.cc:504] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:29.315069 6840 raft_consensus.cc:2454] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 1.
I20240502 21:27:29.315276 6709 raft_consensus.cc:2454] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 1.
I20240502 21:27:29.315488 6513 leader_election.cc:302] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:29.315732 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec2b1b80644a93b9239600157ad957" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:29.315841 7942 raft_consensus.cc:2781] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:29.315920 6970 raft_consensus.cc:3037] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:29.315981 7942 raft_consensus.cc:686] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [term 1 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:29.316074 7892 leader_election.cc:288] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 1 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:29.316288 7942 consensus_queue.cc:237] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:29.316733 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec2b1b80644a93b9239600157ad957" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:29.316968 6840 raft_consensus.cc:3037] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:29.319276 6970 raft_consensus.cc:2454] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 1.
I20240502 21:27:29.320524 6514 leader_election.cc:302] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2; no voters:
I20240502 21:27:29.321264 6840 raft_consensus.cc:2454] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 1.
I20240502 21:27:29.321745 7942 raft_consensus.cc:2781] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:29.321964 7942 raft_consensus.cc:686] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 1 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:29.321913 6437 catalog_manager.cc:5484] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 0 to 1, leader changed from <none> to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 1 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:29.322201 7942 consensus_queue.cc:237] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:29.326598 6437 catalog_manager.cc:5484] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 0 to 1, leader changed from <none> to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 1 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:29.590415 7602 raft_consensus.cc:482] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:29.590566 7602 raft_consensus.cc:504] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:29.591207 7602 leader_election.cc:288] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:29.591599 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2e443bad344ce89ce69871406c5a29" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bc1a0939fe49348a18d3da18f9feae" is_pre_election: true
I20240502 21:27:29.591638 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2e443bad344ce89ce69871406c5a29" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9793130a8e4f45388357b8d2e9f47720" is_pre_election: true
I20240502 21:27:29.591835 6840 raft_consensus.cc:2454] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 0.
I20240502 21:27:29.591857 6579 raft_consensus.cc:2454] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 0.
I20240502 21:27:29.592206 6904 leader_election.cc:302] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:29.592478 7602 raft_consensus.cc:2781] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:29.592636 7602 raft_consensus.cc:482] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:29.592748 7602 raft_consensus.cc:3037] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:29.594579 7602 raft_consensus.cc:504] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:29.595070 7602 leader_election.cc:288] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 1 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:29.595556 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2e443bad344ce89ce69871406c5a29" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:29.595809 6579 raft_consensus.cc:3037] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:29.595736 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2e443bad344ce89ce69871406c5a29" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:29.596714 6840 raft_consensus.cc:3037] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:29.598598 6579 raft_consensus.cc:2454] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 1.
I20240502 21:27:29.598996 6904 leader_election.cc:302] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2; no voters:
I20240502 21:27:29.599308 7602 raft_consensus.cc:2781] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:29.599481 6840 raft_consensus.cc:2454] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 1.
I20240502 21:27:29.599647 7602 raft_consensus.cc:686] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:29.599992 7602 consensus_queue.cc:237] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:29.602656 6437 catalog_manager.cc:5484] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 1 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:29.766939 7942 consensus_queue.cc:1035] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:29.773288 7892 consensus_queue.cc:1035] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:29.796614 7892 consensus_queue.cc:1035] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:29.801885 7760 consensus_queue.cc:1035] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:30.008883 8016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:30.009159 8016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:30.016060 8016 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240502 21:27:30.137210 7602 consensus_queue.cc:1035] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:30.146642 8058 consensus_queue.cc:1035] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:30.254341 6513 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.226.131:33929, user_credentials={real_user=slave}} blocked reactor thread for 55865.2us
INSERT report
rows total: 2000
time total: 230.213 ms
time per row: 0.115107 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 21:27:31.248593 6430 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48454:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:31.256908 6689 tablet_service.cc:1467] Processing CreateTablet for tablet 533d0913d0bc4582b976a58bda8176bc (DEFAULT_TABLE table=rebalance_test_table_6 [id=5d96742d45dd4f359a30233ffa3b107a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:31.257361 6689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 533d0913d0bc4582b976a58bda8176bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:31.257545 6950 tablet_service.cc:1467] Processing CreateTablet for tablet 444ccd95dc0f4bdd806ca9151d56d5cd (DEFAULT_TABLE table=rebalance_test_table_6 [id=5d96742d45dd4f359a30233ffa3b107a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:31.257928 6950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 444ccd95dc0f4bdd806ca9151d56d5cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:31.257862 6559 tablet_service.cc:1467] Processing CreateTablet for tablet 533d0913d0bc4582b976a58bda8176bc (DEFAULT_TABLE table=rebalance_test_table_6 [id=5d96742d45dd4f359a30233ffa3b107a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:31.258262 6559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 533d0913d0bc4582b976a58bda8176bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:31.262252 8085 tablet_bootstrap.cc:492] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae: Bootstrap starting.
I20240502 21:27:31.263617 6691 tablet_service.cc:1467] Processing CreateTablet for tablet 444ccd95dc0f4bdd806ca9151d56d5cd (DEFAULT_TABLE table=rebalance_test_table_6 [id=5d96742d45dd4f359a30233ffa3b107a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:31.263778 8085 tablet_bootstrap.cc:654] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:31.263748 6690 tablet_service.cc:1467] Processing CreateTablet for tablet c9e728880af44303a975a63a4bd7bc62 (DEFAULT_TABLE table=rebalance_test_table_6 [id=5d96742d45dd4f359a30233ffa3b107a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:31.263967 6691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 444ccd95dc0f4bdd806ca9151d56d5cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:31.264240 6690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9e728880af44303a975a63a4bd7bc62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:31.265336 6820 tablet_service.cc:1467] Processing CreateTablet for tablet 444ccd95dc0f4bdd806ca9151d56d5cd (DEFAULT_TABLE table=rebalance_test_table_6 [id=5d96742d45dd4f359a30233ffa3b107a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:31.265758 6820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 444ccd95dc0f4bdd806ca9151d56d5cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:31.268465 6949 tablet_service.cc:1467] Processing CreateTablet for tablet 533d0913d0bc4582b976a58bda8176bc (DEFAULT_TABLE table=rebalance_test_table_6 [id=5d96742d45dd4f359a30233ffa3b107a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:31.268946 6949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 533d0913d0bc4582b976a58bda8176bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:31.277806 8086 tablet_bootstrap.cc:492] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef: Bootstrap starting.
I20240502 21:27:31.279032 8085 tablet_bootstrap.cc:492] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae: No bootstrap required, opened a new log
I20240502 21:27:31.279212 8085 ts_tablet_manager.cc:1367] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.011s
I20240502 21:27:31.279337 6951 tablet_service.cc:1467] Processing CreateTablet for tablet c9e728880af44303a975a63a4bd7bc62 (DEFAULT_TABLE table=rebalance_test_table_6 [id=5d96742d45dd4f359a30233ffa3b107a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:31.279670 6951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9e728880af44303a975a63a4bd7bc62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:31.279989 8085 raft_consensus.cc:348] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:31.280197 8085 raft_consensus.cc:374] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:31.280292 8085 raft_consensus.cc:729] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Initialized, Role: FOLLOWER
I20240502 21:27:31.280545 8085 consensus_queue.cc:260] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:31.281109 8085 ts_tablet_manager.cc:1398] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:31.284058 6558 tablet_service.cc:1467] Processing CreateTablet for tablet c9e728880af44303a975a63a4bd7bc62 (DEFAULT_TABLE table=rebalance_test_table_6 [id=5d96742d45dd4f359a30233ffa3b107a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:31.284427 6558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9e728880af44303a975a63a4bd7bc62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:31.288100 8087 tablet_bootstrap.cc:492] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2: Bootstrap starting.
I20240502 21:27:31.290657 8089 tablet_bootstrap.cc:492] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720: Bootstrap starting.
I20240502 21:27:31.290953 8087 tablet_bootstrap.cc:654] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:31.291251 8085 tablet_bootstrap.cc:492] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae: Bootstrap starting.
I20240502 21:27:31.292191 8089 tablet_bootstrap.cc:654] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:31.292882 8085 tablet_bootstrap.cc:654] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:31.293411 8087 tablet_bootstrap.cc:492] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2: No bootstrap required, opened a new log
I20240502 21:27:31.293576 8087 ts_tablet_manager.cc:1367] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240502 21:27:31.294127 8089 tablet_bootstrap.cc:492] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720: No bootstrap required, opened a new log
I20240502 21:27:31.294252 8089 ts_tablet_manager.cc:1367] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:27:31.294226 8087 raft_consensus.cc:348] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:31.294411 8087 raft_consensus.cc:374] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:31.294498 8087 raft_consensus.cc:729] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Initialized, Role: FOLLOWER
I20240502 21:27:31.294710 8087 consensus_queue.cc:260] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:31.294831 8085 tablet_bootstrap.cc:492] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae: No bootstrap required, opened a new log
I20240502 21:27:31.294968 8089 raft_consensus.cc:348] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:31.295127 8087 ts_tablet_manager.cc:1398] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 21:27:31.295173 8089 raft_consensus.cc:374] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:31.295176 8085 ts_tablet_manager.cc:1367] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:27:31.295297 8089 raft_consensus.cc:729] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Initialized, Role: FOLLOWER
I20240502 21:27:31.295442 8087 tablet_bootstrap.cc:492] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2: Bootstrap starting.
I20240502 21:27:31.295571 8089 consensus_queue.cc:260] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:31.296144 8089 ts_tablet_manager.cc:1398] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:31.296860 8087 tablet_bootstrap.cc:654] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:31.297536 8085 raft_consensus.cc:348] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:31.300410 8062 raft_consensus.cc:482] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:31.300590 8062 raft_consensus.cc:504] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:31.301259 8062 leader_election.cc:288] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:31.301733 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533d0913d0bc4582b976a58bda8176bc" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" is_pre_election: true
I20240502 21:27:31.301821 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533d0913d0bc4582b976a58bda8176bc" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a838c00b544bf68ec4cc9c513212b2" is_pre_election: true
W20240502 21:27:31.302470 6514 leader_election.cc:341] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:31.302870 8085 raft_consensus.cc:374] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:31.303030 8085 raft_consensus.cc:729] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Initialized, Role: FOLLOWER
I20240502 21:27:31.303279 8085 consensus_queue.cc:260] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:31.303653 8085 ts_tablet_manager.cc:1398] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240502 21:27:31.304563 8087 tablet_bootstrap.cc:492] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2: No bootstrap required, opened a new log
I20240502 21:27:31.304718 8087 ts_tablet_manager.cc:1367] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.004s
I20240502 21:27:31.305341 8087 raft_consensus.cc:348] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:31.305554 8087 raft_consensus.cc:374] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:31.305634 8087 raft_consensus.cc:729] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Initialized, Role: FOLLOWER
I20240502 21:27:31.305843 8087 consensus_queue.cc:260] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:31.306326 8087 ts_tablet_manager.cc:1398] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:31.306560 8087 tablet_bootstrap.cc:492] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2: Bootstrap starting.
I20240502 21:27:31.308110 8087 tablet_bootstrap.cc:654] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:31.308554 8086 tablet_bootstrap.cc:654] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:31.309698 8087 tablet_bootstrap.cc:492] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2: No bootstrap required, opened a new log
I20240502 21:27:31.309844 8087 ts_tablet_manager.cc:1367] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
W20240502 21:27:31.310074 6515 leader_election.cc:341] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:31.310189 6515 leader_election.cc:302] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae; no voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:31.310532 8062 raft_consensus.cc:2726] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:27:31.310494 8087 raft_consensus.cc:348] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:31.310674 8087 raft_consensus.cc:374] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:31.310760 8087 raft_consensus.cc:729] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Initialized, Role: FOLLOWER
I20240502 21:27:31.310920 8087 consensus_queue.cc:260] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:31.311436 8087 ts_tablet_manager.cc:1398] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:27:31.311455 8086 tablet_bootstrap.cc:492] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef: No bootstrap required, opened a new log
I20240502 21:27:31.311586 8086 ts_tablet_manager.cc:1367] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
I20240502 21:27:31.312157 8086 raft_consensus.cc:348] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:31.312304 8086 raft_consensus.cc:374] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:31.312386 8086 raft_consensus.cc:729] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Initialized, Role: FOLLOWER
I20240502 21:27:31.312599 8086 consensus_queue.cc:260] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:31.312994 8086 ts_tablet_manager.cc:1398] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240502 21:27:31.313251 8086 tablet_bootstrap.cc:492] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef: Bootstrap starting.
I20240502 21:27:31.314697 8086 tablet_bootstrap.cc:654] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:31.316288 8086 tablet_bootstrap.cc:492] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef: No bootstrap required, opened a new log
I20240502 21:27:31.316761 8086 ts_tablet_manager.cc:1367] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 21:27:31.317476 8086 raft_consensus.cc:348] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:31.317744 8086 raft_consensus.cc:374] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:31.317819 8086 raft_consensus.cc:729] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Initialized, Role: FOLLOWER
I20240502 21:27:31.318089 8086 consensus_queue.cc:260] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:31.318524 8086 ts_tablet_manager.cc:1398] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:27:31.318754 8086 tablet_bootstrap.cc:492] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef: Bootstrap starting.
I20240502 21:27:31.319952 8086 tablet_bootstrap.cc:654] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:31.321916 8086 tablet_bootstrap.cc:492] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef: No bootstrap required, opened a new log
I20240502 21:27:31.322022 8086 ts_tablet_manager.cc:1367] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:31.322546 8086 raft_consensus.cc:348] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:31.322734 8086 raft_consensus.cc:374] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:31.322809 8086 raft_consensus.cc:729] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Initialized, Role: FOLLOWER
I20240502 21:27:31.322962 8086 consensus_queue.cc:260] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:31.323402 8086 ts_tablet_manager.cc:1398] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 21:27:31.389328 8057 raft_consensus.cc:482] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:31.389469 8057 raft_consensus.cc:504] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:31.389989 8057 leader_election.cc:288] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:31.390436 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e728880af44303a975a63a4bd7bc62" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" is_pre_election: true
I20240502 21:27:31.390439 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e728880af44303a975a63a4bd7bc62" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bc1a0939fe49348a18d3da18f9feae" is_pre_election: true
I20240502 21:27:31.390640 6709 raft_consensus.cc:2454] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 0.
I20240502 21:27:31.390669 6579 raft_consensus.cc:2454] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 0.
I20240502 21:27:31.391041 6906 leader_election.cc:302] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:31.391306 8057 raft_consensus.cc:2781] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:31.391412 8057 raft_consensus.cc:482] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:31.391474 8057 raft_consensus.cc:3037] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:31.393564 8057 raft_consensus.cc:504] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:31.394049 8057 leader_election.cc:288] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 1 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:31.394379 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e728880af44303a975a63a4bd7bc62" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:31.394479 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e728880af44303a975a63a4bd7bc62" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:31.394538 6709 raft_consensus.cc:3037] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:31.394649 6579 raft_consensus.cc:3037] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:31.396426 6709 raft_consensus.cc:2454] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 1.
I20240502 21:27:31.396692 6579 raft_consensus.cc:2454] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 1.
I20240502 21:27:31.396803 6906 leader_election.cc:302] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:31.397119 8057 raft_consensus.cc:2781] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:31.397503 8057 raft_consensus.cc:686] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:31.397789 8057 consensus_queue.cc:237] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:31.400215 6430 catalog_manager.cc:5484] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 1 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:31.411351 7642 raft_consensus.cc:482] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:31.411470 7642 raft_consensus.cc:504] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:31.412070 7642 leader_election.cc:288] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:31.412437 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533d0913d0bc4582b976a58bda8176bc" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bc1a0939fe49348a18d3da18f9feae" is_pre_election: true
I20240502 21:27:31.412549 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533d0913d0bc4582b976a58bda8176bc" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a838c00b544bf68ec4cc9c513212b2" is_pre_election: true
I20240502 21:27:31.412664 6579 raft_consensus.cc:2454] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 0.
I20240502 21:27:31.412762 6970 raft_consensus.cc:2454] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 0.
I20240502 21:27:31.413030 6644 leader_election.cc:302] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:31.413295 7642 raft_consensus.cc:2781] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:31.413421 7642 raft_consensus.cc:482] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:31.413509 7642 raft_consensus.cc:3037] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:31.415210 7642 raft_consensus.cc:504] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:31.415719 7642 leader_election.cc:288] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 1 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:31.416067 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533d0913d0bc4582b976a58bda8176bc" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:31.416277 6579 raft_consensus.cc:3037] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:31.416267 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533d0913d0bc4582b976a58bda8176bc" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:31.416456 6970 raft_consensus.cc:3037] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:31.418217 6579 raft_consensus.cc:2454] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 1.
I20240502 21:27:31.418231 6970 raft_consensus.cc:2454] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 1.
I20240502 21:27:31.418553 6645 leader_election.cc:302] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:31.418907 7642 raft_consensus.cc:2781] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:31.419291 7642 raft_consensus.cc:686] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:31.419557 7642 consensus_queue.cc:237] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:31.422228 6430 catalog_manager.cc:5484] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 0 to 1, leader changed from <none> to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 1 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:31.515197 8057 raft_consensus.cc:482] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:31.515410 8057 raft_consensus.cc:504] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:31.515969 8057 leader_election.cc:288] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:31.516371 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444ccd95dc0f4bdd806ca9151d56d5cd" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" is_pre_election: true
I20240502 21:27:31.516453 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444ccd95dc0f4bdd806ca9151d56d5cd" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9793130a8e4f45388357b8d2e9f47720" is_pre_election: true
I20240502 21:27:31.516655 6841 raft_consensus.cc:2454] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 0.
I20240502 21:27:31.516929 6709 raft_consensus.cc:2454] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 0.
I20240502 21:27:31.517066 6904 leader_election.cc:302] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:31.517328 8057 raft_consensus.cc:2781] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:31.517417 8057 raft_consensus.cc:482] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:31.517529 8057 raft_consensus.cc:3037] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:31.519304 8057 raft_consensus.cc:504] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:31.519846 8057 leader_election.cc:288] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 1 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:31.520186 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444ccd95dc0f4bdd806ca9151d56d5cd" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:31.520291 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444ccd95dc0f4bdd806ca9151d56d5cd" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:31.520371 6709 raft_consensus.cc:3037] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:31.520498 6841 raft_consensus.cc:3037] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:31.522178 6709 raft_consensus.cc:2454] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 1.
I20240502 21:27:31.522568 6906 leader_election.cc:302] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:31.522722 6841 raft_consensus.cc:2454] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 1.
I20240502 21:27:31.522785 8057 raft_consensus.cc:2781] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:31.522940 8057 raft_consensus.cc:686] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 1 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:31.523180 8057 consensus_queue.cc:237] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:31.526072 6430 catalog_manager.cc:5484] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 1 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:31.799818 8057 consensus_queue.cc:1035] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:31.805970 8107 consensus_queue.cc:1035] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:31.879205 8104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:31.879457 8104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:31.886209 8104 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240502 21:27:31.964541 8133 consensus_queue.cc:1035] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:31.965015 7642 consensus_queue.cc:1035] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:32.019627 6840 raft_consensus.cc:1261] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Refusing update from remote peer 80a838c00b544bf68ec4cc9c513212b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:32.027526 8107 consensus_queue.cc:1035] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:32.034360 6709 raft_consensus.cc:1261] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Refusing update from remote peer 80a838c00b544bf68ec4cc9c513212b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:32.037932 8107 consensus_queue.cc:1035] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:32.162434 8150 mvcc.cc:205] Tried to move back new op lower bound from 7023350792265297920 to 7023350790241542144. Current Snapshot: MvccSnapshot[applied={T|T < 7023350792265297920 or (T in {7023350792265297920})}]
I20240502 21:27:32.179173 8152 mvcc.cc:205] Tried to move back new op lower bound from 7023350792335208448 to 7023350790241542144. Current Snapshot: MvccSnapshot[applied={T|T < 7023350792335208448}]
INSERT report
rows total: 2000
time total: 260.878 ms
time per row: 0.130439 ms
I20240502 21:27:32.276296 6430 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48454:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:32.284956 6691 tablet_service.cc:1467] Processing CreateTablet for tablet da14dcc58d684704b3649e85b26fb43c (DEFAULT_TABLE table=rebalance_test_table_7 [id=4826c8c047224642a8fab14d7e3574ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:32.285099 6820 tablet_service.cc:1467] Processing CreateTablet for tablet da14dcc58d684704b3649e85b26fb43c (DEFAULT_TABLE table=rebalance_test_table_7 [id=4826c8c047224642a8fab14d7e3574ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:32.285434 6691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da14dcc58d684704b3649e85b26fb43c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:32.285460 6820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da14dcc58d684704b3649e85b26fb43c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:32.286615 6950 tablet_service.cc:1467] Processing CreateTablet for tablet c10bca264dda4180bd19beb5767bf047 (DEFAULT_TABLE table=rebalance_test_table_7 [id=4826c8c047224642a8fab14d7e3574ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:32.287051 6950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c10bca264dda4180bd19beb5767bf047. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:32.286952 6558 tablet_service.cc:1467] Processing CreateTablet for tablet da14dcc58d684704b3649e85b26fb43c (DEFAULT_TABLE table=rebalance_test_table_7 [id=4826c8c047224642a8fab14d7e3574ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:32.287397 6558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da14dcc58d684704b3649e85b26fb43c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:32.288419 6949 tablet_service.cc:1467] Processing CreateTablet for tablet 3b6b526ec34c41aea64a2a19f0208f5d (DEFAULT_TABLE table=rebalance_test_table_7 [id=4826c8c047224642a8fab14d7e3574ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:32.288746 6949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b6b526ec34c41aea64a2a19f0208f5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:32.291294 6559 tablet_service.cc:1467] Processing CreateTablet for tablet 3b6b526ec34c41aea64a2a19f0208f5d (DEFAULT_TABLE table=rebalance_test_table_7 [id=4826c8c047224642a8fab14d7e3574ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:32.291668 6559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b6b526ec34c41aea64a2a19f0208f5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:32.293880 6821 tablet_service.cc:1467] Processing CreateTablet for tablet 3b6b526ec34c41aea64a2a19f0208f5d (DEFAULT_TABLE table=rebalance_test_table_7 [id=4826c8c047224642a8fab14d7e3574ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:32.294193 6821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b6b526ec34c41aea64a2a19f0208f5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:32.296303 6689 tablet_service.cc:1467] Processing CreateTablet for tablet c10bca264dda4180bd19beb5767bf047 (DEFAULT_TABLE table=rebalance_test_table_7 [id=4826c8c047224642a8fab14d7e3574ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:32.296737 6689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c10bca264dda4180bd19beb5767bf047. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:32.296955 6819 tablet_service.cc:1467] Processing CreateTablet for tablet c10bca264dda4180bd19beb5767bf047 (DEFAULT_TABLE table=rebalance_test_table_7 [id=4826c8c047224642a8fab14d7e3574ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:32.297281 6819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c10bca264dda4180bd19beb5767bf047. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:32.299710 8170 tablet_bootstrap.cc:492] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae: Bootstrap starting.
I20240502 21:27:32.304639 8170 tablet_bootstrap.cc:654] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:32.304939 8169 tablet_bootstrap.cc:492] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720: Bootstrap starting.
I20240502 21:27:32.305763 8172 tablet_bootstrap.cc:492] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef: Bootstrap starting.
I20240502 21:27:32.307718 8172 tablet_bootstrap.cc:654] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:32.308025 8169 tablet_bootstrap.cc:654] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:32.313472 8170 tablet_bootstrap.cc:492] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae: No bootstrap required, opened a new log
I20240502 21:27:32.313648 8170 ts_tablet_manager.cc:1367] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240502 21:27:32.316404 8170 raft_consensus.cc:348] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:32.316967 8170 raft_consensus.cc:374] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:32.317240 8170 raft_consensus.cc:729] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Initialized, Role: FOLLOWER
I20240502 21:27:32.317633 8170 consensus_queue.cc:260] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:32.318084 8171 tablet_bootstrap.cc:492] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2: Bootstrap starting.
I20240502 21:27:32.318710 8170 ts_tablet_manager.cc:1398] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:27:32.318979 8170 tablet_bootstrap.cc:492] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae: Bootstrap starting.
I20240502 21:27:32.320307 8171 tablet_bootstrap.cc:654] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:32.320497 8170 tablet_bootstrap.cc:654] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:32.325896 8169 tablet_bootstrap.cc:492] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720: No bootstrap required, opened a new log
I20240502 21:27:32.326113 8169 ts_tablet_manager.cc:1367] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.009s
I20240502 21:27:32.327155 8169 raft_consensus.cc:348] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:32.327504 8169 raft_consensus.cc:374] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:32.327667 8169 raft_consensus.cc:729] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Initialized, Role: FOLLOWER
I20240502 21:27:32.328015 8169 consensus_queue.cc:260] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:32.328941 8172 tablet_bootstrap.cc:492] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef: No bootstrap required, opened a new log
I20240502 21:27:32.328948 8169 ts_tablet_manager.cc:1398] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:27:32.329164 8172 ts_tablet_manager.cc:1367] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.005s
I20240502 21:27:32.329243 8169 tablet_bootstrap.cc:492] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720: Bootstrap starting.
I20240502 21:27:32.330026 8172 raft_consensus.cc:348] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:32.330283 8172 raft_consensus.cc:374] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:32.330430 8172 raft_consensus.cc:729] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Initialized, Role: FOLLOWER
I20240502 21:27:32.330731 8172 consensus_queue.cc:260] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:32.331038 8169 tablet_bootstrap.cc:654] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:32.331625 8172 ts_tablet_manager.cc:1398] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:32.331977 8172 tablet_bootstrap.cc:492] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef: Bootstrap starting.
I20240502 21:27:32.333214 8169 tablet_bootstrap.cc:492] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720: No bootstrap required, opened a new log
I20240502 21:27:32.333425 8169 ts_tablet_manager.cc:1367] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:27:32.334681 8169 raft_consensus.cc:348] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:32.334825 8170 tablet_bootstrap.cc:492] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae: No bootstrap required, opened a new log
I20240502 21:27:32.334882 8169 raft_consensus.cc:374] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:32.335004 8170 ts_tablet_manager.cc:1367] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.006s
I20240502 21:27:32.335009 8169 raft_consensus.cc:729] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Initialized, Role: FOLLOWER
I20240502 21:27:32.335091 8172 tablet_bootstrap.cc:654] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:32.335294 8169 consensus_queue.cc:260] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:32.335865 8169 ts_tablet_manager.cc:1398] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:32.335851 8170 raft_consensus.cc:348] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:32.336082 8170 raft_consensus.cc:374] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:32.336171 8170 raft_consensus.cc:729] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Initialized, Role: FOLLOWER
I20240502 21:27:32.336258 8169 tablet_bootstrap.cc:492] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720: Bootstrap starting.
I20240502 21:27:32.336395 8170 consensus_queue.cc:260] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:32.338477 8170 ts_tablet_manager.cc:1398] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:32.340178 8171 tablet_bootstrap.cc:492] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2: No bootstrap required, opened a new log
I20240502 21:27:32.340512 8171 ts_tablet_manager.cc:1367] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2: Time spent bootstrapping tablet: real 0.023s user 0.012s sys 0.000s
I20240502 21:27:32.340770 8169 tablet_bootstrap.cc:654] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:32.342190 8171 raft_consensus.cc:348] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:32.342407 8171 raft_consensus.cc:374] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:32.342505 8171 raft_consensus.cc:729] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Initialized, Role: FOLLOWER
I20240502 21:27:32.342608 8169 tablet_bootstrap.cc:492] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720: No bootstrap required, opened a new log
I20240502 21:27:32.342746 8169 ts_tablet_manager.cc:1367] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 21:27:32.342723 8171 consensus_queue.cc:260] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:32.343272 8171 ts_tablet_manager.cc:1398] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:32.343403 8169 raft_consensus.cc:348] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:32.343571 8171 tablet_bootstrap.cc:492] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2: Bootstrap starting.
I20240502 21:27:32.343572 8169 raft_consensus.cc:374] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:32.343650 8169 raft_consensus.cc:729] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Initialized, Role: FOLLOWER
I20240502 21:27:32.343837 8169 consensus_queue.cc:260] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:32.344913 8169 ts_tablet_manager.cc:1398] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:32.344934 8171 tablet_bootstrap.cc:654] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:32.347043 8171 tablet_bootstrap.cc:492] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2: No bootstrap required, opened a new log
I20240502 21:27:32.347249 8171 ts_tablet_manager.cc:1367] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:27:32.348251 8172 tablet_bootstrap.cc:492] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef: No bootstrap required, opened a new log
I20240502 21:27:32.348426 8172 ts_tablet_manager.cc:1367] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.003s
I20240502 21:27:32.348148 8171 raft_consensus.cc:348] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:32.349062 8172 raft_consensus.cc:348] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:32.349237 8172 raft_consensus.cc:374] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:32.349274 8171 raft_consensus.cc:374] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:32.349319 8172 raft_consensus.cc:729] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Initialized, Role: FOLLOWER
I20240502 21:27:32.349395 8171 raft_consensus.cc:729] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Initialized, Role: FOLLOWER
I20240502 21:27:32.349570 8172 consensus_queue.cc:260] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:32.349632 8171 consensus_queue.cc:260] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:32.350065 8172 ts_tablet_manager.cc:1398] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:32.350247 8171 ts_tablet_manager.cc:1398] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:27:32.399369 8173 raft_consensus.cc:482] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:32.399498 8173 raft_consensus.cc:504] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:32.400039 8173 leader_election.cc:288] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:32.400409 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da14dcc58d684704b3649e85b26fb43c" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" is_pre_election: true
I20240502 21:27:32.400471 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da14dcc58d684704b3649e85b26fb43c" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bc1a0939fe49348a18d3da18f9feae" is_pre_election: true
I20240502 21:27:32.400625 6579 raft_consensus.cc:2454] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 0.
I20240502 21:27:32.400636 6711 raft_consensus.cc:2454] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 0.
I20240502 21:27:32.401036 6774 leader_election.cc:302] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:32.401293 8173 raft_consensus.cc:2781] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:32.401402 8173 raft_consensus.cc:482] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:32.401487 8173 raft_consensus.cc:3037] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:32.403200 8173 raft_consensus.cc:504] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:32.403620 8173 leader_election.cc:288] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:32.403947 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da14dcc58d684704b3649e85b26fb43c" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:32.404117 6711 raft_consensus.cc:3037] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:32.404054 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da14dcc58d684704b3649e85b26fb43c" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:32.404234 6579 raft_consensus.cc:3037] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:32.405781 6711 raft_consensus.cc:2454] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 1.
I20240502 21:27:32.405882 6579 raft_consensus.cc:2454] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 1.
I20240502 21:27:32.406093 6776 leader_election.cc:302] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:32.406317 8173 raft_consensus.cc:2781] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:32.406423 8173 raft_consensus.cc:686] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 1 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:32.406649 8173 consensus_queue.cc:237] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:32.409284 6431 catalog_manager.cc:5484] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 1 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:32.444566 8062 raft_consensus.cc:482] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:32.444708 8062 raft_consensus.cc:504] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:32.445237 8062 leader_election.cc:288] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:32.445544 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6b526ec34c41aea64a2a19f0208f5d" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9793130a8e4f45388357b8d2e9f47720" is_pre_election: true
I20240502 21:27:32.445590 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6b526ec34c41aea64a2a19f0208f5d" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a838c00b544bf68ec4cc9c513212b2" is_pre_election: true
I20240502 21:27:32.445722 6840 raft_consensus.cc:2454] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 0.
I20240502 21:27:32.445791 6970 raft_consensus.cc:2454] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 0.
I20240502 21:27:32.446064 6513 leader_election.cc:302] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:32.446334 8062 raft_consensus.cc:2781] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:32.446470 8062 raft_consensus.cc:482] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:32.446550 8062 raft_consensus.cc:3037] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:32.448426 8062 raft_consensus.cc:504] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:32.449003 8062 leader_election.cc:288] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 1 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:32.449271 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6b526ec34c41aea64a2a19f0208f5d" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:32.449476 6840 raft_consensus.cc:3037] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:32.449460 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6b526ec34c41aea64a2a19f0208f5d" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:32.449661 6970 raft_consensus.cc:3037] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:32.451258 6970 raft_consensus.cc:2454] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 1.
I20240502 21:27:32.451273 6840 raft_consensus.cc:2454] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 1.
I20240502 21:27:32.451608 6514 leader_election.cc:302] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2; no voters:
I20240502 21:27:32.451874 8062 raft_consensus.cc:2781] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:32.452216 8062 raft_consensus.cc:686] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 1 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:32.452484 8062 consensus_queue.cc:237] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:32.454949 6431 catalog_manager.cc:5484] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 0 to 1, leader changed from <none> to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 1 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:32.565124 8173 raft_consensus.cc:482] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:32.565310 8173 raft_consensus.cc:504] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:32.565831 8173 leader_election.cc:288] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:32.566202 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10bca264dda4180bd19beb5767bf047" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a838c00b544bf68ec4cc9c513212b2" is_pre_election: true
I20240502 21:27:32.566256 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10bca264dda4180bd19beb5767bf047" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" is_pre_election: true
I20240502 21:27:32.566382 6970 raft_consensus.cc:2454] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 0.
I20240502 21:27:32.566447 6711 raft_consensus.cc:2454] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 0.
I20240502 21:27:32.566727 6775 leader_election.cc:302] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:32.567010 8173 raft_consensus.cc:2781] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:32.567102 8173 raft_consensus.cc:482] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:32.567210 8173 raft_consensus.cc:3037] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:32.569041 8173 raft_consensus.cc:504] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:32.569504 8173 leader_election.cc:288] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:32.569862 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10bca264dda4180bd19beb5767bf047" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:32.569924 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10bca264dda4180bd19beb5767bf047" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:32.570029 6970 raft_consensus.cc:3037] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:32.570101 6711 raft_consensus.cc:3037] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:32.571708 6970 raft_consensus.cc:2454] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 1.
I20240502 21:27:32.571789 6711 raft_consensus.cc:2454] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 1.
I20240502 21:27:32.572074 6775 leader_election.cc:302] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:32.572355 8173 raft_consensus.cc:2781] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:32.572523 8173 raft_consensus.cc:686] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 1 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:32.572767 8173 consensus_queue.cc:237] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:32.575711 6431 catalog_manager.cc:5484] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 1 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
W20240502 21:27:32.888659 8187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:32.888931 8187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:32.895308 8187 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240502 21:27:32.922797 8062 consensus_queue.cc:1035] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:32.929826 8062 consensus_queue.cc:1035] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:32.993873 8173 consensus_queue.cc:1035] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:32.999692 8173 consensus_queue.cc:1035] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:33.088600 7937 consensus_queue.cc:1035] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:33.119228 6709 raft_consensus.cc:1261] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Refusing update from remote peer 9793130a8e4f45388357b8d2e9f47720: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:33.120811 8173 consensus_queue.cc:1035] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 291.921 ms
time per row: 0.145961 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 21:27:34.194129 6437 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48454:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:34.201951 6691 tablet_service.cc:1467] Processing CreateTablet for tablet 647ab470df6f491a81d040a1647ca95b (DEFAULT_TABLE table=rebalance_test_table_8 [id=13c0be4c5fea4e758657200904e2f495]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:34.202239 6558 tablet_service.cc:1467] Processing CreateTablet for tablet 647ab470df6f491a81d040a1647ca95b (DEFAULT_TABLE table=rebalance_test_table_8 [id=13c0be4c5fea4e758657200904e2f495]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:34.202385 6691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 647ab470df6f491a81d040a1647ca95b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:34.202597 6558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 647ab470df6f491a81d040a1647ca95b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:34.202986 6950 tablet_service.cc:1467] Processing CreateTablet for tablet aae0bc634aa7424e935c8bc81a85099b (DEFAULT_TABLE table=rebalance_test_table_8 [id=13c0be4c5fea4e758657200904e2f495]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:34.203291 6559 tablet_service.cc:1467] Processing CreateTablet for tablet aae0bc634aa7424e935c8bc81a85099b (DEFAULT_TABLE table=rebalance_test_table_8 [id=13c0be4c5fea4e758657200904e2f495]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:34.203395 6950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae0bc634aa7424e935c8bc81a85099b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:34.203644 6559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae0bc634aa7424e935c8bc81a85099b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:34.204505 6821 tablet_service.cc:1467] Processing CreateTablet for tablet 647ab470df6f491a81d040a1647ca95b (DEFAULT_TABLE table=rebalance_test_table_8 [id=13c0be4c5fea4e758657200904e2f495]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:34.204869 6821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 647ab470df6f491a81d040a1647ca95b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:34.209074 6949 tablet_service.cc:1467] Processing CreateTablet for tablet 3619e8a307de4583a4e470ad583fd7bc (DEFAULT_TABLE table=rebalance_test_table_8 [id=13c0be4c5fea4e758657200904e2f495]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:34.209422 6949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3619e8a307de4583a4e470ad583fd7bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:34.209939 6689 tablet_service.cc:1467] Processing CreateTablet for tablet aae0bc634aa7424e935c8bc81a85099b (DEFAULT_TABLE table=rebalance_test_table_8 [id=13c0be4c5fea4e758657200904e2f495]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:34.210286 6689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae0bc634aa7424e935c8bc81a85099b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:34.213207 6819 tablet_service.cc:1467] Processing CreateTablet for tablet 3619e8a307de4583a4e470ad583fd7bc (DEFAULT_TABLE table=rebalance_test_table_8 [id=13c0be4c5fea4e758657200904e2f495]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:34.213590 6819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3619e8a307de4583a4e470ad583fd7bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:34.216823 6560 tablet_service.cc:1467] Processing CreateTablet for tablet 3619e8a307de4583a4e470ad583fd7bc (DEFAULT_TABLE table=rebalance_test_table_8 [id=13c0be4c5fea4e758657200904e2f495]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:34.217298 6560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3619e8a307de4583a4e470ad583fd7bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:34.217604 8260 tablet_bootstrap.cc:492] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef: Bootstrap starting.
I20240502 21:27:34.219936 8260 tablet_bootstrap.cc:654] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:34.221163 8261 tablet_bootstrap.cc:492] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae: Bootstrap starting.
I20240502 21:27:34.224380 8261 tablet_bootstrap.cc:654] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:34.225230 8260 tablet_bootstrap.cc:492] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef: No bootstrap required, opened a new log
I20240502 21:27:34.225404 8260 ts_tablet_manager.cc:1367] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 21:27:34.226084 8260 raft_consensus.cc:348] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:34.226267 8260 raft_consensus.cc:374] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:34.226387 8260 raft_consensus.cc:729] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Initialized, Role: FOLLOWER
I20240502 21:27:34.226699 8260 consensus_queue.cc:260] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:34.228358 8260 ts_tablet_manager.cc:1398] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:27:34.228703 8260 tablet_bootstrap.cc:492] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef: Bootstrap starting.
I20240502 21:27:34.232635 8260 tablet_bootstrap.cc:654] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:34.233283 8262 tablet_bootstrap.cc:492] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2: Bootstrap starting.
I20240502 21:27:34.234629 8262 tablet_bootstrap.cc:654] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:34.234894 8249 raft_consensus.cc:482] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:34.235055 8249 raft_consensus.cc:504] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:34.235993 8249 leader_election.cc:288] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:34.236828 8265 tablet_bootstrap.cc:492] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720: Bootstrap starting.
I20240502 21:27:34.238497 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bc1a0939fe49348a18d3da18f9feae" is_pre_election: true
I20240502 21:27:34.239365 8265 tablet_bootstrap.cc:654] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:34.244536 8260 tablet_bootstrap.cc:492] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef: No bootstrap required, opened a new log
I20240502 21:27:34.244920 8260 ts_tablet_manager.cc:1367] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.004s
I20240502 21:27:34.246083 8260 raft_consensus.cc:348] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:34.247530 8260 raft_consensus.cc:374] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:34.247861 8260 raft_consensus.cc:729] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Initialized, Role: FOLLOWER
W20240502 21:27:34.248973 6644 leader_election.cc:341] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:34.249399 8260 consensus_queue.cc:260] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:34.253383 8260 ts_tablet_manager.cc:1398] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240502 21:27:34.260679 8261 tablet_bootstrap.cc:492] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae: No bootstrap required, opened a new log
I20240502 21:27:34.260845 8261 ts_tablet_manager.cc:1367] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.002s
I20240502 21:27:34.260913 8262 tablet_bootstrap.cc:492] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2: No bootstrap required, opened a new log
I20240502 21:27:34.261082 8262 ts_tablet_manager.cc:1367] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.021s
I20240502 21:27:34.261579 8261 raft_consensus.cc:348] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:34.261791 8261 raft_consensus.cc:374] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:34.261891 8261 raft_consensus.cc:729] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Initialized, Role: FOLLOWER
I20240502 21:27:34.261945 8262 raft_consensus.cc:348] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:34.262184 8262 raft_consensus.cc:374] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:34.262131 8261 consensus_queue.cc:260] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:34.262288 8262 raft_consensus.cc:729] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Initialized, Role: FOLLOWER
I20240502 21:27:34.262722 8261 ts_tablet_manager.cc:1398] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:27:34.262967 8262 consensus_queue.cc:260] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:34.263737 8265 tablet_bootstrap.cc:492] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720: No bootstrap required, opened a new log
I20240502 21:27:34.263948 8265 ts_tablet_manager.cc:1367] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.013s
I20240502 21:27:34.264293 8262 ts_tablet_manager.cc:1398] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:34.264596 8262 tablet_bootstrap.cc:492] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2: Bootstrap starting.
I20240502 21:27:34.264811 8265 raft_consensus.cc:348] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:34.265045 8265 raft_consensus.cc:374] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:34.265174 8265 raft_consensus.cc:729] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Initialized, Role: FOLLOWER
I20240502 21:27:34.265446 8265 consensus_queue.cc:260] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:34.266224 8261 tablet_bootstrap.cc:492] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae: Bootstrap starting.
I20240502 21:27:34.266698 8262 tablet_bootstrap.cc:654] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:34.266835 8265 ts_tablet_manager.cc:1398] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:27:34.267211 8265 tablet_bootstrap.cc:492] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720: Bootstrap starting.
I20240502 21:27:34.269285 8265 tablet_bootstrap.cc:654] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:34.274820 8261 tablet_bootstrap.cc:654] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:34.276537 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9793130a8e4f45388357b8d2e9f47720" is_pre_election: true
I20240502 21:27:34.276741 6841 raft_consensus.cc:2454] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 0.
I20240502 21:27:34.277204 6644 leader_election.cc:302] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 51bc1a0939fe49348a18d3da18f9feae
I20240502 21:27:34.277660 8249 raft_consensus.cc:2781] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:34.277766 8249 raft_consensus.cc:482] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:34.277841 8249 raft_consensus.cc:3037] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:34.280381 8249 raft_consensus.cc:504] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:34.280928 8249 leader_election.cc:288] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 1 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:34.281594 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:34.281764 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:34.281960 6841 raft_consensus.cc:3037] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:27:34.282212 6644 leader_election.cc:341] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:27:34.284361 6841 raft_consensus.cc:2454] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 1.
I20240502 21:27:34.284710 6644 leader_election.cc:302] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 51bc1a0939fe49348a18d3da18f9feae
I20240502 21:27:34.285239 8249 raft_consensus.cc:2781] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:34.286902 8261 tablet_bootstrap.cc:492] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae: No bootstrap required, opened a new log
I20240502 21:27:34.287201 8261 ts_tablet_manager.cc:1367] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20240502 21:27:34.288206 8261 raft_consensus.cc:348] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:34.288412 8261 raft_consensus.cc:374] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:34.288508 8261 raft_consensus.cc:729] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Initialized, Role: FOLLOWER
I20240502 21:27:34.288738 8261 consensus_queue.cc:260] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:34.289450 8261 ts_tablet_manager.cc:1398] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:34.289870 8261 tablet_bootstrap.cc:492] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae: Bootstrap starting.
I20240502 21:27:34.290660 8265 tablet_bootstrap.cc:492] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720: No bootstrap required, opened a new log
I20240502 21:27:34.290899 8265 ts_tablet_manager.cc:1367] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.017s
I20240502 21:27:34.292901 8265 raft_consensus.cc:348] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:34.293237 8265 raft_consensus.cc:374] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:34.293362 8265 raft_consensus.cc:729] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Initialized, Role: FOLLOWER
I20240502 21:27:34.293898 8265 consensus_queue.cc:260] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:34.294726 8188 raft_consensus.cc:482] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:34.294904 8261 tablet_bootstrap.cc:654] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:34.294895 8188 raft_consensus.cc:504] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:34.295610 8188 leader_election.cc:288] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:34.296252 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3619e8a307de4583a4e470ad583fd7bc" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9793130a8e4f45388357b8d2e9f47720" is_pre_election: true
I20240502 21:27:34.296415 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3619e8a307de4583a4e470ad583fd7bc" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bc1a0939fe49348a18d3da18f9feae" is_pre_election: true
I20240502 21:27:34.296496 6841 raft_consensus.cc:2454] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 0.
I20240502 21:27:34.296672 6579 raft_consensus.cc:2454] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 0.
I20240502 21:27:34.297147 6904 leader_election.cc:302] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2; no voters:
I20240502 21:27:34.297264 8265 ts_tablet_manager.cc:1398] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 21:27:34.297475 8188 raft_consensus.cc:2781] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:34.297631 8188 raft_consensus.cc:482] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:34.297739 8188 raft_consensus.cc:3037] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:34.298213 8262 tablet_bootstrap.cc:492] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2: No bootstrap required, opened a new log
I20240502 21:27:34.298372 8262 ts_tablet_manager.cc:1367] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.022s
I20240502 21:27:34.299219 8262 raft_consensus.cc:348] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:34.299463 8262 raft_consensus.cc:374] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:34.299539 8262 raft_consensus.cc:729] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Initialized, Role: FOLLOWER
I20240502 21:27:34.299923 8262 consensus_queue.cc:260] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:34.301085 8262 ts_tablet_manager.cc:1398] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:27:34.302861 8188 raft_consensus.cc:504] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:34.303673 8188 leader_election.cc:288] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 1 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:34.304039 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3619e8a307de4583a4e470ad583fd7bc" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:34.304194 6841 raft_consensus.cc:3037] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:34.304189 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3619e8a307de4583a4e470ad583fd7bc" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:34.304384 6579 raft_consensus.cc:3037] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:34.306229 6579 raft_consensus.cc:2454] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 1.
I20240502 21:27:34.306715 6841 raft_consensus.cc:2454] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 1.
I20240502 21:27:34.306808 6904 leader_election.cc:302] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2; no voters:
I20240502 21:27:34.307152 8249 raft_consensus.cc:686] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:34.307565 8188 raft_consensus.cc:2781] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:34.307705 8188 raft_consensus.cc:686] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 1 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:34.307590 8249 consensus_queue.cc:237] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:34.307972 8188 consensus_queue.cc:237] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:34.309567 8261 tablet_bootstrap.cc:492] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae: No bootstrap required, opened a new log
I20240502 21:27:34.309759 8261 ts_tablet_manager.cc:1367] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.001s
I20240502 21:27:34.310456 8261 raft_consensus.cc:348] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:34.310638 8261 raft_consensus.cc:374] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:34.310719 8261 raft_consensus.cc:729] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Initialized, Role: FOLLOWER
I20240502 21:27:34.310554 6437 catalog_manager.cc:5484] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 1 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:34.310912 8261 consensus_queue.cc:260] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:34.311391 8261 ts_tablet_manager.cc:1398] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:34.313880 6432 catalog_manager.cc:5484] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 0 to 1, leader changed from <none> to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 1 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:34.403841 8249 raft_consensus.cc:482] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:34.404011 8249 raft_consensus.cc:504] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:34.404646 8249 leader_election.cc:288] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:34.404984 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae0bc634aa7424e935c8bc81a85099b" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bc1a0939fe49348a18d3da18f9feae" is_pre_election: true
I20240502 21:27:34.405041 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae0bc634aa7424e935c8bc81a85099b" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a838c00b544bf68ec4cc9c513212b2" is_pre_election: true
I20240502 21:27:34.405162 6579 raft_consensus.cc:2454] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 0.
I20240502 21:27:34.405279 6970 raft_consensus.cc:2454] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 0.
I20240502 21:27:34.405534 6644 leader_election.cc:302] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:34.405802 8249 raft_consensus.cc:2781] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:34.405875 8249 raft_consensus.cc:482] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:34.405957 8249 raft_consensus.cc:3037] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:34.407694 8249 raft_consensus.cc:504] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:34.408200 8249 leader_election.cc:288] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 1 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:34.408586 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae0bc634aa7424e935c8bc81a85099b" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:34.408612 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae0bc634aa7424e935c8bc81a85099b" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:34.408802 6970 raft_consensus.cc:3037] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:34.408800 6579 raft_consensus.cc:3037] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:34.411438 6970 raft_consensus.cc:2454] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 1.
I20240502 21:27:34.411453 6579 raft_consensus.cc:2454] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 1.
I20240502 21:27:34.411783 6645 leader_election.cc:302] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:34.412024 8249 raft_consensus.cc:2781] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:34.412166 8249 raft_consensus.cc:686] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:34.412389 8249 consensus_queue.cc:237] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:34.415174 6431 catalog_manager.cc:5484] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 0 to 1, leader changed from <none> to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 1 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:34.523694 8062 raft_consensus.cc:482] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:34.523906 8062 raft_consensus.cc:504] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:34.524592 8062 leader_election.cc:288] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:34.524991 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" is_pre_election: true
I20240502 21:27:34.525035 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9793130a8e4f45388357b8d2e9f47720" is_pre_election: true
I20240502 21:27:34.525388 6841 raft_consensus.cc:2379] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae in current term 1: Already voted for candidate ab5c055dc24f4944a58d7d2593ddc8ef in this term.
I20240502 21:27:34.525812 6513 leader_election.cc:302] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae; no voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:34.526106 8062 raft_consensus.cc:3037] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:34.528016 8062 raft_consensus.cc:2726] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:27:34.747496 8249 consensus_queue.cc:1035] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:34.754426 8249 consensus_queue.cc:1035] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:34.763602 8188 consensus_queue.cc:1035] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:34.785185 8276 consensus_queue.cc:1035] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:34.793398 8280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:34.793751 8280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:34.801267 8280 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240502 21:27:34.972714 6580 raft_consensus.cc:1261] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Refusing update from remote peer ab5c055dc24f4944a58d7d2593ddc8ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:34.973389 6970 raft_consensus.cc:1261] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Refusing update from remote peer ab5c055dc24f4944a58d7d2593ddc8ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:34.974045 8284 consensus_queue.cc:1035] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:34.974717 8284 consensus_queue.cc:1035] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 215.246 ms
time per row: 0.107623 ms
I20240502 21:27:35.154696 6437 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48454:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:35.162853 6691 tablet_service.cc:1467] Processing CreateTablet for tablet 6b2b3e30b94d47dd822539cb30f36f81 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a9cc1b07c2124a37a902488f4a8e3dd7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:35.163293 6691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b2b3e30b94d47dd822539cb30f36f81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:35.163195 6949 tablet_service.cc:1467] Processing CreateTablet for tablet 6b2b3e30b94d47dd822539cb30f36f81 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a9cc1b07c2124a37a902488f4a8e3dd7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:35.163439 6689 tablet_service.cc:1467] Processing CreateTablet for tablet d3093485f30b41939a1380ab7aa9e38b (DEFAULT_TABLE table=rebalance_test_table_9 [id=a9cc1b07c2124a37a902488f4a8e3dd7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:35.163596 6949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b2b3e30b94d47dd822539cb30f36f81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:35.163844 6689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3093485f30b41939a1380ab7aa9e38b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:35.166625 6690 tablet_service.cc:1467] Processing CreateTablet for tablet df917e7196ed4939b75dc52930c87640 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a9cc1b07c2124a37a902488f4a8e3dd7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:35.166971 6690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df917e7196ed4939b75dc52930c87640. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:35.167140 6950 tablet_service.cc:1467] Processing CreateTablet for tablet d3093485f30b41939a1380ab7aa9e38b (DEFAULT_TABLE table=rebalance_test_table_9 [id=a9cc1b07c2124a37a902488f4a8e3dd7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:35.167528 6950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3093485f30b41939a1380ab7aa9e38b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:35.167793 6821 tablet_service.cc:1467] Processing CreateTablet for tablet 6b2b3e30b94d47dd822539cb30f36f81 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a9cc1b07c2124a37a902488f4a8e3dd7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:35.168215 6821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b2b3e30b94d47dd822539cb30f36f81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:35.169822 6819 tablet_service.cc:1467] Processing CreateTablet for tablet d3093485f30b41939a1380ab7aa9e38b (DEFAULT_TABLE table=rebalance_test_table_9 [id=a9cc1b07c2124a37a902488f4a8e3dd7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:35.170189 6819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3093485f30b41939a1380ab7aa9e38b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:35.172799 6820 tablet_service.cc:1467] Processing CreateTablet for tablet df917e7196ed4939b75dc52930c87640 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a9cc1b07c2124a37a902488f4a8e3dd7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:35.173147 6820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df917e7196ed4939b75dc52930c87640. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:35.178997 6558 tablet_service.cc:1467] Processing CreateTablet for tablet df917e7196ed4939b75dc52930c87640 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a9cc1b07c2124a37a902488f4a8e3dd7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:35.179422 6558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df917e7196ed4939b75dc52930c87640. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:35.180545 8347 tablet_bootstrap.cc:492] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720: Bootstrap starting.
I20240502 21:27:35.182374 8346 tablet_bootstrap.cc:492] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2: Bootstrap starting.
I20240502 21:27:35.183682 8347 tablet_bootstrap.cc:654] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:35.184120 8346 tablet_bootstrap.cc:654] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:35.190722 8348 tablet_bootstrap.cc:492] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef: Bootstrap starting.
I20240502 21:27:35.190994 8351 tablet_bootstrap.cc:492] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae: Bootstrap starting.
I20240502 21:27:35.192497 8348 tablet_bootstrap.cc:654] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:35.193581 8351 tablet_bootstrap.cc:654] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:35.193976 8347 tablet_bootstrap.cc:492] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720: No bootstrap required, opened a new log
I20240502 21:27:35.193976 8346 tablet_bootstrap.cc:492] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2: No bootstrap required, opened a new log
I20240502 21:27:35.194128 8346 ts_tablet_manager.cc:1367] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240502 21:27:35.194152 8347 ts_tablet_manager.cc:1367] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20240502 21:27:35.194852 8346 raft_consensus.cc:348] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:35.194852 8347 raft_consensus.cc:348] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:35.195040 8346 raft_consensus.cc:374] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:35.195132 8346 raft_consensus.cc:729] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Initialized, Role: FOLLOWER
I20240502 21:27:35.195153 8347 raft_consensus.cc:374] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:35.195331 8347 raft_consensus.cc:729] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Initialized, Role: FOLLOWER
I20240502 21:27:35.195366 8346 consensus_queue.cc:260] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:35.195631 8347 consensus_queue.cc:260] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:35.195868 8346 ts_tablet_manager.cc:1398] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:35.196132 8346 tablet_bootstrap.cc:492] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2: Bootstrap starting.
I20240502 21:27:35.196291 8347 ts_tablet_manager.cc:1398] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:27:35.196640 8347 tablet_bootstrap.cc:492] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720: Bootstrap starting.
I20240502 21:27:35.198158 8347 tablet_bootstrap.cc:654] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:35.201565 8346 tablet_bootstrap.cc:654] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:35.206132 8347 tablet_bootstrap.cc:492] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720: No bootstrap required, opened a new log
I20240502 21:27:35.206373 8347 ts_tablet_manager.cc:1367] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20240502 21:27:35.206729 8348 tablet_bootstrap.cc:492] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef: No bootstrap required, opened a new log
I20240502 21:27:35.206861 8348 ts_tablet_manager.cc:1367] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.004s
I20240502 21:27:35.207274 8347 raft_consensus.cc:348] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:35.207458 8348 raft_consensus.cc:348] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:35.207562 8347 raft_consensus.cc:374] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:35.207644 8348 raft_consensus.cc:374] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:35.207645 8347 raft_consensus.cc:729] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Initialized, Role: FOLLOWER
I20240502 21:27:35.207739 8348 raft_consensus.cc:729] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Initialized, Role: FOLLOWER
I20240502 21:27:35.207966 8348 consensus_queue.cc:260] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:35.207955 8347 consensus_queue.cc:260] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:35.208550 8348 ts_tablet_manager.cc:1398] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:35.208626 8347 ts_tablet_manager.cc:1398] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:27:35.208856 8348 tablet_bootstrap.cc:492] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef: Bootstrap starting.
I20240502 21:27:35.208890 8347 tablet_bootstrap.cc:492] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720: Bootstrap starting.
I20240502 21:27:35.210198 8348 tablet_bootstrap.cc:654] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:35.210487 8347 tablet_bootstrap.cc:654] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:35.211469 8351 tablet_bootstrap.cc:492] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae: No bootstrap required, opened a new log
I20240502 21:27:35.211591 8351 ts_tablet_manager.cc:1367] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.003s
I20240502 21:27:35.212334 8351 raft_consensus.cc:348] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:35.212491 8346 tablet_bootstrap.cc:492] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2: No bootstrap required, opened a new log
I20240502 21:27:35.212523 8351 raft_consensus.cc:374] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:35.212610 8351 raft_consensus.cc:729] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Initialized, Role: FOLLOWER
I20240502 21:27:35.212625 8346 ts_tablet_manager.cc:1367] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.005s
I20240502 21:27:35.212788 8351 consensus_queue.cc:260] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:35.213235 8347 tablet_bootstrap.cc:492] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720: No bootstrap required, opened a new log
I20240502 21:27:35.213276 8351 ts_tablet_manager.cc:1398] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:35.213343 8347 ts_tablet_manager.cc:1367] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:27:35.214413 8346 raft_consensus.cc:348] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:35.214602 8346 raft_consensus.cc:374] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:35.214663 8346 raft_consensus.cc:729] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Initialized, Role: FOLLOWER
I20240502 21:27:35.214581 8347 raft_consensus.cc:348] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:35.214738 8347 raft_consensus.cc:374] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:35.214814 8347 raft_consensus.cc:729] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Initialized, Role: FOLLOWER
I20240502 21:27:35.214990 8347 consensus_queue.cc:260] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:35.215039 8346 consensus_queue.cc:260] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:35.215508 8347 ts_tablet_manager.cc:1398] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 21:27:35.216116 8346 ts_tablet_manager.cc:1398] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:35.217141 8348 tablet_bootstrap.cc:492] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef: No bootstrap required, opened a new log
I20240502 21:27:35.217298 8348 ts_tablet_manager.cc:1367] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s
I20240502 21:27:35.218017 8348 raft_consensus.cc:348] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:35.218233 8348 raft_consensus.cc:374] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:35.218331 8348 raft_consensus.cc:729] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Initialized, Role: FOLLOWER
I20240502 21:27:35.218530 8348 consensus_queue.cc:260] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:35.219008 8348 ts_tablet_manager.cc:1398] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:35.219259 8348 tablet_bootstrap.cc:492] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef: Bootstrap starting.
I20240502 21:27:35.220608 8348 tablet_bootstrap.cc:654] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:35.222373 8348 tablet_bootstrap.cc:492] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef: No bootstrap required, opened a new log
I20240502 21:27:35.222489 8348 ts_tablet_manager.cc:1367] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:27:35.223008 8348 raft_consensus.cc:348] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:35.223132 8348 raft_consensus.cc:374] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:35.223207 8348 raft_consensus.cc:729] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Initialized, Role: FOLLOWER
I20240502 21:27:35.223385 8348 consensus_queue.cc:260] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:35.223752 8348 ts_tablet_manager.cc:1398] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:27:35.303490 8281 raft_consensus.cc:482] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:35.303635 8281 raft_consensus.cc:504] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:35.304237 8281 leader_election.cc:288] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:35.304540 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2b3e30b94d47dd822539cb30f36f81" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" is_pre_election: true
I20240502 21:27:35.304651 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2b3e30b94d47dd822539cb30f36f81" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a838c00b544bf68ec4cc9c513212b2" is_pre_election: true
I20240502 21:27:35.304754 6709 raft_consensus.cc:2454] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 0.
I20240502 21:27:35.304862 6970 raft_consensus.cc:2454] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 0.
I20240502 21:27:35.305099 6776 leader_election.cc:302] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:35.305297 8281 raft_consensus.cc:2781] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:35.305393 8281 raft_consensus.cc:482] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:35.305469 8281 raft_consensus.cc:3037] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:35.307296 8281 raft_consensus.cc:504] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:35.307857 8281 leader_election.cc:288] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:35.308223 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2b3e30b94d47dd822539cb30f36f81" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:35.308450 6709 raft_consensus.cc:3037] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:35.308491 6971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2b3e30b94d47dd822539cb30f36f81" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:35.308667 6971 raft_consensus.cc:3037] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:35.311094 6709 raft_consensus.cc:2454] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 1.
I20240502 21:27:35.311193 6971 raft_consensus.cc:2454] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 1.
I20240502 21:27:35.311545 6775 leader_election.cc:302] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:35.311815 8281 raft_consensus.cc:2781] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:35.311931 8281 raft_consensus.cc:686] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 1 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:35.312199 8281 consensus_queue.cc:237] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:35.314857 6433 catalog_manager.cc:5484] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 1 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:35.363698 8274 raft_consensus.cc:482] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:35.363830 8274 raft_consensus.cc:504] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:35.364454 8274 leader_election.cc:288] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:35.364681 6971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3093485f30b41939a1380ab7aa9e38b" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a838c00b544bf68ec4cc9c513212b2" is_pre_election: true
I20240502 21:27:35.364883 6971 raft_consensus.cc:2454] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 0.
I20240502 21:27:35.364859 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3093485f30b41939a1380ab7aa9e38b" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9793130a8e4f45388357b8d2e9f47720" is_pre_election: true
I20240502 21:27:35.365036 6840 raft_consensus.cc:2454] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 0.
I20240502 21:27:35.365231 6645 leader_election.cc:302] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:35.365610 8274 raft_consensus.cc:2781] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:35.365700 8274 raft_consensus.cc:482] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:35.365774 8274 raft_consensus.cc:3037] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:35.367707 8274 raft_consensus.cc:504] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:35.368204 8274 leader_election.cc:288] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 1 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:35.368609 6971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3093485f30b41939a1380ab7aa9e38b" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:35.368777 6971 raft_consensus.cc:3037] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:35.368777 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3093485f30b41939a1380ab7aa9e38b" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:35.368965 6840 raft_consensus.cc:3037] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:35.370628 6971 raft_consensus.cc:2454] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 1.
I20240502 21:27:35.370829 6840 raft_consensus.cc:2454] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 1.
I20240502 21:27:35.370911 6645 leader_election.cc:302] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:35.371274 8274 raft_consensus.cc:2781] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:35.371393 8274 raft_consensus.cc:686] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:35.371608 8274 consensus_queue.cc:237] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:35.373947 6433 catalog_manager.cc:5484] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 0 to 1, leader changed from <none> to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 1 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:35.402290 8274 raft_consensus.cc:482] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:35.402410 8274 raft_consensus.cc:504] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:35.402809 8274 leader_election.cc:288] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:35.403128 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df917e7196ed4939b75dc52930c87640" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bc1a0939fe49348a18d3da18f9feae" is_pre_election: true
I20240502 21:27:35.403227 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df917e7196ed4939b75dc52930c87640" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9793130a8e4f45388357b8d2e9f47720" is_pre_election: true
I20240502 21:27:35.403364 6580 raft_consensus.cc:2454] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 0.
I20240502 21:27:35.403405 6840 raft_consensus.cc:2454] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 0.
I20240502 21:27:35.403708 6644 leader_election.cc:302] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:35.403952 8274 raft_consensus.cc:2781] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:35.404031 8274 raft_consensus.cc:482] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:35.404125 8274 raft_consensus.cc:3037] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:35.405741 8274 raft_consensus.cc:504] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:35.406158 8274 leader_election.cc:288] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 1 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:35.406525 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df917e7196ed4939b75dc52930c87640" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:35.406574 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df917e7196ed4939b75dc52930c87640" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:35.406740 6840 raft_consensus.cc:3037] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:35.406740 6580 raft_consensus.cc:3037] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:35.409574 6840 raft_consensus.cc:2454] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 1.
I20240502 21:27:35.409636 6580 raft_consensus.cc:2454] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 1.
I20240502 21:27:35.409998 6644 leader_election.cc:302] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:35.410269 8274 raft_consensus.cc:2781] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:35.410452 8274 raft_consensus.cc:686] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:35.410686 8274 consensus_queue.cc:237] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:35.413190 6433 catalog_manager.cc:5484] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 0 to 1, leader changed from <none> to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 1 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
W20240502 21:27:35.738333 8362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:35.738595 8362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:35.744830 8362 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240502 21:27:35.825803 8274 consensus_queue.cc:1035] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:35.826987 8281 consensus_queue.cc:1035] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:35.869772 8385 consensus_queue.cc:1035] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:35.891330 8284 consensus_queue.cc:1035] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:35.919498 8398 consensus_queue.cc:1035] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:35.927079 6580 raft_consensus.cc:1261] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Refusing update from remote peer ab5c055dc24f4944a58d7d2593ddc8ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:35.927765 8398 consensus_queue.cc:1035] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 257.206 ms
time per row: 0.128603 ms
I20240502 21:27:36.130267 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.133:42877
--local_ip_for_outbound_sockets=127.24.226.133
--tserver_master_addrs=127.24.226.190:40767
--webserver_port=39743
--webserver_interface=127.24.226.133
--builtin_ntp_servers=127.24.226.148:46251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 21:27:36.452944 8423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:36.453260 8423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:27:36.453395 8423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:36.460923 8423 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:27:36.461042 8423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:27:36.461184 8423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.133
I20240502 21:27:36.469534 8423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:46251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.133:42877
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.24.226.133
--webserver_port=39743
--tserver_master_addrs=127.24.226.190:40767
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.133
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:27:36.470681 8423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:27:36.472216 8423 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:36.484005 8429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:36.484040 8431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:36.483979 8428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:36.485080 8423 server_base.cc:1034] running on GCE node
I20240502 21:27:36.485584 8423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:27:36.486356 8423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:27:36.487566 8423 hybrid_clock.cc:648] HybridClock initialized: now 1714685256487519 us; error 37 us; skew 500 ppm
I20240502 21:27:36.487859 8423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:36.490018 8423 webserver.cc:460] Webserver started at http://127.24.226.133:39743/ using document root <none> and password file <none>
I20240502 21:27:36.490672 8423 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:36.490794 8423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:36.495803 8423 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 21:27:36.498452 8437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:36.499572 8423 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:36.499720 8423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "6676d476ceb44beba1c2de8395f4627d"
format_stamp: "Formatted at 2024-05-02 21:27:17 on dist-test-slave-41k1"
I20240502 21:27:36.500537 8423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:36.550627 8423 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:36.551375 8423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:27:36.551633 8423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:36.552572 8423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:36.554247 8423 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:36.554334 8423 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:36.554462 8423 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:36.554517 8423 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:36.586819 8423 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.133:42877
I20240502 21:27:36.586887 8550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.133:42877 every 8 connection(s)
I20240502 21:27:36.588248 8423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:27:36.591756 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 8423
I20240502 21:27:36.592525 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.134:36267
--local_ip_for_outbound_sockets=127.24.226.134
--tserver_master_addrs=127.24.226.190:40767
--webserver_port=44171
--webserver_interface=127.24.226.134
--builtin_ntp_servers=127.24.226.148:46251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:27:36.604033 8551 heartbeater.cc:347] Connected to a master server at 127.24.226.190:40767
I20240502 21:27:36.604367 8551 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:36.605384 8551 heartbeater.cc:511] Master 127.24.226.190:40767 requested a full tablet report, sending...
I20240502 21:27:36.607069 6433 ts_manager.cc:194] Re-registered known tserver with Master: 6676d476ceb44beba1c2de8395f4627d (127.24.226.133:42877)
I20240502 21:27:36.608348 6433 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.133:39479
W20240502 21:27:36.923811 8555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:36.924080 8555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:27:36.924198 8555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:36.931154 8555 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:27:36.931317 8555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:27:36.931458 8555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.134
I20240502 21:27:36.939721 8555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:46251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.134:36267
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.24.226.134
--webserver_port=44171
--tserver_master_addrs=127.24.226.190:40767
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.134
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:27:36.940817 8555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:27:36.942744 8555 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:36.953629 8562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:36.953605 8564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:36.953656 8561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:36.954557 8555 server_base.cc:1034] running on GCE node
I20240502 21:27:36.955046 8555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:27:36.955861 8555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:27:36.957096 8555 hybrid_clock.cc:648] HybridClock initialized: now 1714685256957005 us; error 71 us; skew 500 ppm
I20240502 21:27:36.957388 8555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:36.960129 8555 webserver.cc:460] Webserver started at http://127.24.226.134:44171/ using document root <none> and password file <none>
I20240502 21:27:36.960740 8555 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:36.960855 8555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:36.965166 8555 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:27:36.967468 8570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:36.968700 8555 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 21:27:36.968847 8555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "e329c23b07f34d4597def2ad455b9c0c"
format_stamp: "Formatted at 2024-05-02 21:27:17 on dist-test-slave-41k1"
I20240502 21:27:36.969548 8555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:36.999650 8555 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:37.000383 8555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:27:37.000690 8555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:37.001643 8555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:37.003163 8555 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:37.003254 8555 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:37.003374 8555 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:37.003428 8555 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:37.035890 8555 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.134:36267
I20240502 21:27:37.035940 8682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.134:36267 every 8 connection(s)
I20240502 21:27:37.037577 8555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 21:27:37.038250 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 8555
I20240502 21:27:37.039070 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.135:43095
--local_ip_for_outbound_sockets=127.24.226.135
--tserver_master_addrs=127.24.226.190:40767
--webserver_port=33877
--webserver_interface=127.24.226.135
--builtin_ntp_servers=127.24.226.148:46251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:27:37.052203 8683 heartbeater.cc:347] Connected to a master server at 127.24.226.190:40767
I20240502 21:27:37.052505 8683 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:37.053283 8683 heartbeater.cc:511] Master 127.24.226.190:40767 requested a full tablet report, sending...
I20240502 21:27:37.054679 6433 ts_manager.cc:194] Re-registered known tserver with Master: e329c23b07f34d4597def2ad455b9c0c (127.24.226.134:36267)
I20240502 21:27:37.055799 6433 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.134:46625
W20240502 21:27:37.367869 8686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:37.368216 8686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:27:37.368371 8686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:37.375011 8686 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:27:37.375103 8686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:27:37.375201 8686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.135
I20240502 21:27:37.383554 8686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:46251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.135:43095
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.24.226.135
--webserver_port=33877
--tserver_master_addrs=127.24.226.190:40767
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.135
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:27:37.384640 8686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:27:37.386271 8686 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:37.398128 8692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:37.398176 8693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:37.398340 8695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:37.399276 8686 server_base.cc:1034] running on GCE node
I20240502 21:27:37.399793 8686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:27:37.400682 8686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:27:37.401875 8686 hybrid_clock.cc:648] HybridClock initialized: now 1714685257401831 us; error 37 us; skew 500 ppm
I20240502 21:27:37.402173 8686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:37.404220 8686 webserver.cc:460] Webserver started at http://127.24.226.135:33877/ using document root <none> and password file <none>
I20240502 21:27:37.404939 8686 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:37.405061 8686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:37.409780 8686 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20240502 21:27:37.412444 8701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:37.413636 8686 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 21:27:37.413780 8686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "34f4e21e024e464b9450592a5d9a6c8b"
format_stamp: "Formatted at 2024-05-02 21:27:18 on dist-test-slave-41k1"
I20240502 21:27:37.414525 8686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:37.432521 8686 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:37.433187 8686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:27:37.433434 8686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:37.434219 8686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:37.435801 8686 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:37.435887 8686 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:37.436012 8686 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:37.436077 8686 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:37.466450 8686 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.135:43095
I20240502 21:27:37.466502 8814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.135:43095 every 8 connection(s)
I20240502 21:27:37.467681 8686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 21:27:37.472990 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 8686
I20240502 21:27:37.473539 25482 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD89o3N/build/asan/bin/kudu
/tmp/dist-test-taskD89o3N/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.226.136:45511
--local_ip_for_outbound_sockets=127.24.226.136
--tserver_master_addrs=127.24.226.190:40767
--webserver_port=46577
--webserver_interface=127.24.226.136
--builtin_ntp_servers=127.24.226.148:46251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:27:37.479341 8815 heartbeater.cc:347] Connected to a master server at 127.24.226.190:40767
I20240502 21:27:37.479629 8815 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:37.480608 8815 heartbeater.cc:511] Master 127.24.226.190:40767 requested a full tablet report, sending...
I20240502 21:27:37.482643 6433 ts_manager.cc:194] Re-registered known tserver with Master: 34f4e21e024e464b9450592a5d9a6c8b (127.24.226.135:43095)
I20240502 21:27:37.483395 6433 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.135:55241
I20240502 21:27:37.611191 8551 heartbeater.cc:503] Master 127.24.226.190:40767 was elected leader, sending a full tablet report...
W20240502 21:27:37.785527 8818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:37.785761 8818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:27:37.785873 8818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:37.792675 8818 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:27:37.792783 8818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:27:37.792915 8818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.226.136
I20240502 21:27:37.800801 8818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.226.148:46251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.226.136:45511
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.24.226.136
--webserver_port=46577
--tserver_master_addrs=127.24.226.190:40767
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.226.136
--log_dir=/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:27:37.801791 8818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:27:37.803447 8818 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:37.814357 8826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:37.814790 8828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:37.814893 8825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:37.817016 8818 server_base.cc:1034] running on GCE node
I20240502 21:27:37.817734 8818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:27:37.819190 8818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:27:37.820410 8818 hybrid_clock.cc:648] HybridClock initialized: now 1714685257820325 us; error 80 us; skew 500 ppm
I20240502 21:27:37.820727 8818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:37.823240 8818 webserver.cc:460] Webserver started at http://127.24.226.136:46577/ using document root <none> and password file <none>
I20240502 21:27:37.824124 8818 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:37.824262 8818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:37.829133 8818 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20240502 21:27:37.831612 8834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:37.832779 8818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 21:27:37.832908 8818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "8a9bbbab70e444c18da6ad8f7a2c290a"
format_stamp: "Formatted at 2024-05-02 21:27:18 on dist-test-slave-41k1"
I20240502 21:27:37.833561 8818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:37.861225 8818 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:37.861855 8818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:27:37.862085 8818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:37.862869 8818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:37.864256 8818 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:37.864326 8818 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:37.864406 8818 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:37.864467 8818 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20240502 21:27:37.893260 8818 rpc_server.cc:307] RPC server started. Bound to: 127.24.226.136:45511
I20240502 21:27:37.893329 8947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.226.136:45511 every 8 connection(s)
I20240502 21:27:37.894495 8818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD89o3N/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714685121724517-25482-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 21:27:37.895902 25482 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD89o3N/build/asan/bin/kudu as pid 8818
I20240502 21:27:37.911563 8948 heartbeater.cc:347] Connected to a master server at 127.24.226.190:40767
I20240502 21:27:37.911847 8948 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:37.912756 8948 heartbeater.cc:511] Master 127.24.226.190:40767 requested a full tablet report, sending...
I20240502 21:27:37.914125 6433 ts_manager.cc:194] Re-registered known tserver with Master: 8a9bbbab70e444c18da6ad8f7a2c290a (127.24.226.136:45511)
I20240502 21:27:37.915313 6433 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.226.136:43359
I20240502 21:27:37.967242 6579 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "df917e7196ed4939b75dc52930c87640"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:37.967523 6579 raft_consensus.cc:482] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:37.967635 6579 raft_consensus.cc:3037] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:37.969735 6579 raft_consensus.cc:504] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:37.970288 6579 leader_election.cc:288] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 2 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:37.970729 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df917e7196ed4939b75dc52930c87640" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:37.970763 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df917e7196ed4939b75dc52930c87640" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:37.970938 6840 raft_consensus.cc:3037] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:37.970932 6711 raft_consensus.cc:3032] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:37.971035 6711 raft_consensus.cc:729] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:37.971359 6711 consensus_queue.cc:260] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:37.971889 6711 raft_consensus.cc:3037] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:37.973474 6840 raft_consensus.cc:2454] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 2.
I20240502 21:27:37.973891 6513 leader_election.cc:302] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:37.974064 6579 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:37.974259 6579 raft_consensus.cc:482] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:37.974391 6579 raft_consensus.cc:3037] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:37.974534 8819 raft_consensus.cc:2781] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:37.974671 8819 raft_consensus.cc:686] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:37.974929 8819 consensus_queue.cc:237] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:37.976716 6579 raft_consensus.cc:504] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:37.977277 6711 raft_consensus.cc:2454] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 2.
I20240502 21:27:37.977929 6579 leader_election.cc:288] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 2 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:37.978466 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:37.978658 6709 raft_consensus.cc:3032] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:37.978749 6709 raft_consensus.cc:729] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:37.978940 6709 consensus_queue.cc:260] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:37.979015 6579 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3619e8a307de4583a4e470ad583fd7bc"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:37.979169 6579 raft_consensus.cc:482] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:37.979261 6709 raft_consensus.cc:3037] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:37.979275 6579 raft_consensus.cc:3037] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:37.981561 6579 raft_consensus.cc:504] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:37.981714 6709 raft_consensus.cc:2454] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 2.
I20240502 21:27:37.982224 6515 leader_election.cc:302] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:37.982517 8332 raft_consensus.cc:2781] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:37.982913 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3619e8a307de4583a4e470ad583fd7bc" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:37.983150 6969 raft_consensus.cc:3032] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:37.983103 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3619e8a307de4583a4e470ad583fd7bc" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:37.983254 6969 raft_consensus.cc:729] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:37.983289 6841 raft_consensus.cc:3037] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:37.983561 6969 consensus_queue.cc:260] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:37.984086 6969 raft_consensus.cc:3037] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:37.984509 6579 leader_election.cc:288] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 2 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:37.985275 6580 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aae0bc634aa7424e935c8bc81a85099b"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:37.985426 6580 raft_consensus.cc:482] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:37.985507 6580 raft_consensus.cc:3037] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:37.985661 6841 raft_consensus.cc:2454] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 2.
I20240502 21:27:37.986147 6513 leader_election.cc:302] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:37.986487 8961 raft_consensus.cc:2781] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:37.986492 6969 raft_consensus.cc:2454] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 2.
I20240502 21:27:37.988070 6580 raft_consensus.cc:504] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:37.988601 8332 raft_consensus.cc:686] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:37.988941 8332 consensus_queue.cc:237] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:37.989429 6580 leader_election.cc:288] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 2 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:37.990031 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae0bc634aa7424e935c8bc81a85099b" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:37.990202 6969 raft_consensus.cc:3037] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:37.990309 6580 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3b6b526ec34c41aea64a2a19f0208f5d"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:37.990443 6580 raft_consensus.cc:465] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 21:27:37.990453 6433 catalog_manager.cc:5484] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 1 to 2, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 2 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:37.991268 6579 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "da14dcc58d684704b3649e85b26fb43c"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:37.991405 6579 raft_consensus.cc:482] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:37.991501 6579 raft_consensus.cc:3037] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:37.992794 6969 raft_consensus.cc:2454] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 2.
I20240502 21:27:37.993654 6514 leader_election.cc:302] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2; no voters:
I20240502 21:27:37.993870 6579 raft_consensus.cc:504] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:37.994887 6710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da14dcc58d684704b3649e85b26fb43c" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:37.995110 6710 raft_consensus.cc:3037] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:37.995121 6579 leader_election.cc:288] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 2 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:37.995532 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da14dcc58d684704b3649e85b26fb43c" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:37.995688 6841 raft_consensus.cc:3032] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:37.995765 6841 raft_consensus.cc:729] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:37.995885 6580 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "66198d766e6e4ceb820bc92bcca67d85"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:37.996018 6580 raft_consensus.cc:482] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:37.996017 6841 consensus_queue.cc:260] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:37.996172 6580 raft_consensus.cc:3037] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:37.996570 6841 raft_consensus.cc:3037] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:37.997613 6710 raft_consensus.cc:2454] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 2.
I20240502 21:27:37.998016 6515 leader_election.cc:302] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:37.998445 8819 raft_consensus.cc:2781] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:37.998663 6580 raft_consensus.cc:504] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:37.999163 6580 leader_election.cc:288] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 2 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:37.999612 6841 raft_consensus.cc:2454] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 2.
I20240502 21:27:37.999614 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66198d766e6e4ceb820bc92bcca67d85" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:37.999804 6839 raft_consensus.cc:3037] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:37.999897 6578 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f0b941baaab542a78748d4d0894cfbf4"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:38.000051 6578 raft_consensus.cc:482] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.000151 6578 raft_consensus.cc:3037] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.002395 6578 raft_consensus.cc:504] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.002471 8965 raft_consensus.cc:2781] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:38.002604 6839 raft_consensus.cc:2454] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 2.
I20240502 21:27:38.003393 6578 leader_election.cc:288] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 2 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:38.003764 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66198d766e6e4ceb820bc92bcca67d85" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.003983 6969 raft_consensus.cc:3032] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:38.004069 6969 raft_consensus.cc:729] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.004467 6578 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5e38d544ee14490c9fb5d8069440fd6a"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:38.004365 6969 consensus_queue.cc:260] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:38.004683 6578 raft_consensus.cc:482] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.004765 6578 raft_consensus.cc:3037] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.004781 6969 raft_consensus.cc:3037] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.006791 6578 raft_consensus.cc:504] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.007551 6969 raft_consensus.cc:2454] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 2.
I20240502 21:27:38.007650 6578 leader_election.cc:288] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 2 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:38.007741 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e38d544ee14490c9fb5d8069440fd6a" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.007972 6970 raft_consensus.cc:3037] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.008164 6514 leader_election.cc:302] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2; no voters:
I20240502 21:27:38.008603 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5b2e443bad344ce89ce69871406c5a29"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:38.008785 6577 raft_consensus.cc:482] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.008920 6577 raft_consensus.cc:3037] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.010473 6970 raft_consensus.cc:2454] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 2.
I20240502 21:27:38.011349 6577 raft_consensus.cc:504] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.011907 6577 leader_election.cc:288] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 2 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:38.012835 8967 raft_consensus.cc:2781] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:38.012952 8967 raft_consensus.cc:686] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:38.013569 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e38d544ee14490c9fb5d8069440fd6a" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:38.013685 6838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2e443bad344ce89ce69871406c5a29" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:38.013746 6841 raft_consensus.cc:3032] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:38.013845 6841 raft_consensus.cc:729] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:38.013849 6838 raft_consensus.cc:3037] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.014101 6841 consensus_queue.cc:260] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.014457 6841 raft_consensus.cc:3037] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.016367 6838 raft_consensus.cc:2454] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 2.
I20240502 21:27:38.016942 6514 leader_election.cc:302] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2; no voters:
I20240502 21:27:38.017544 8819 raft_consensus.cc:686] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:38.017527 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2e443bad344ce89ce69871406c5a29" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.017695 6970 raft_consensus.cc:3032] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:38.017779 6970 raft_consensus.cc:729] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.017818 8819 consensus_queue.cc:237] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.018013 6970 consensus_queue.cc:260] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.018371 6970 raft_consensus.cc:3037] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.019410 6513 leader_election.cc:302] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:38.019599 6710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b941baaab542a78748d4d0894cfbf4" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.019829 6710 raft_consensus.cc:3037] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.022727 6710 raft_consensus.cc:2454] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 2.
I20240502 21:27:38.022833 6970 raft_consensus.cc:2454] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 2.
I20240502 21:27:38.023169 6515 leader_election.cc:302] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:38.023160 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b941baaab542a78748d4d0894cfbf4" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:38.023370 6839 raft_consensus.cc:3032] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:38.023456 6839 raft_consensus.cc:729] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:38.023785 6839 consensus_queue.cc:260] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.024188 6839 raft_consensus.cc:3037] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.016844 6841 raft_consensus.cc:2454] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 2.
I20240502 21:27:38.027046 6839 raft_consensus.cc:2454] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 2.
I20240502 21:27:38.027621 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fb023d72aa8d4248a64d4723e18a3cf6"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:38.027854 6577 raft_consensus.cc:482] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.027971 6577 raft_consensus.cc:3037] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:37.982355 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:38.032346 6840 raft_consensus.cc:3037] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.033504 8969 raft_consensus.cc:2781] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:38.033617 8969 raft_consensus.cc:686] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:38.033895 8969 consensus_queue.cc:237] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.034797 6840 raft_consensus.cc:2454] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 2.
I20240502 21:27:38.013370 8967 consensus_queue.cc:237] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:38.039155 8970 raft_consensus.cc:2781] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:38.039850 8965 raft_consensus.cc:686] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:38.039984 8966 raft_consensus.cc:2781] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:38.040189 8965 consensus_queue.cc:237] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:38.041366 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae0bc634aa7424e935c8bc81a85099b" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.041671 6709 raft_consensus.cc:3032] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:38.041841 6709 raft_consensus.cc:729] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:38.042165 6709 consensus_queue.cc:260] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.042652 6709 raft_consensus.cc:3037] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.043865 8966 raft_consensus.cc:686] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:38.044261 8966 consensus_queue.cc:237] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.048965 8970 raft_consensus.cc:686] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:38.049331 8970 consensus_queue.cc:237] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.051327 6709 raft_consensus.cc:2454] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 2.
I20240502 21:27:38.052101 6577 raft_consensus.cc:504] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.053063 6577 leader_election.cc:288] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 2 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:38.053622 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb023d72aa8d4248a64d4723e18a3cf6" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.053838 6969 raft_consensus.cc:3032] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:38.053916 6969 raft_consensus.cc:729] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.054239 6969 consensus_queue.cc:260] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.054563 6969 raft_consensus.cc:3037] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.057341 6969 raft_consensus.cc:2454] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 2.
I20240502 21:27:38.058526 8683 heartbeater.cc:503] Master 127.24.226.190:40767 was elected leader, sending a full tablet report...
I20240502 21:27:38.057871 6514 leader_election.cc:302] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2; no voters:
I20240502 21:27:38.069779 8970 raft_consensus.cc:2781] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:38.069937 8970 raft_consensus.cc:686] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:38.053828 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb023d72aa8d4248a64d4723e18a3cf6" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.070586 6711 raft_consensus.cc:3037] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.071168 8970 consensus_queue.cc:237] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.074033 6711 raft_consensus.cc:2454] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 2.
I20240502 21:27:38.074851 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c6d11cba6ca04c17a9f47506824d9aa2"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:38.076050 6577 raft_consensus.cc:482] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.076511 6577 raft_consensus.cc:3037] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.084360 6577 raft_consensus.cc:504] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.092842 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d11cba6ca04c17a9f47506824d9aa2" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:38.093161 6839 raft_consensus.cc:3032] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:38.093295 6839 raft_consensus.cc:729] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:38.093714 6839 consensus_queue.cc:260] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.094276 6839 raft_consensus.cc:3037] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.097692 6839 raft_consensus.cc:2454] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 2.
I20240502 21:27:38.098738 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d11cba6ca04c17a9f47506824d9aa2" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.098908 6709 raft_consensus.cc:3037] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.098956 6577 leader_election.cc:288] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 2 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:38.100987 6513 leader_election.cc:302] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:38.101289 6709 raft_consensus.cc:2454] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 2.
I20240502 21:27:38.101585 8965 raft_consensus.cc:2781] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:38.101763 8965 raft_consensus.cc:686] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:38.101984 8965 consensus_queue.cc:237] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.101533 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "331bcde3e6bf43c081af836c8cdebdc7"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:38.107493 6577 raft_consensus.cc:465] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 21:27:38.108593 8961 raft_consensus.cc:686] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:38.108628 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fbec2b1b80644a93b9239600157ad957"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:38.108814 6577 raft_consensus.cc:465] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 21:27:38.108911 8961 consensus_queue.cc:237] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:38.109728 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1c98cbfbd0a44335b75b228c217f9132"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:38.109907 6577 raft_consensus.cc:465] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 21:27:38.110625 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "169899da82594d9cb673f8db74764ca2"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:38.110791 6577 raft_consensus.cc:482] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.110898 6577 raft_consensus.cc:3037] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.113340 6577 raft_consensus.cc:504] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.114725 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169899da82594d9cb673f8db74764ca2" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.114928 6709 raft_consensus.cc:3037] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.117323 6709 raft_consensus.cc:2454] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 2.
I20240502 21:27:38.117942 6515 leader_election.cc:302] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:38.118257 8962 raft_consensus.cc:2781] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:38.118386 6577 leader_election.cc:288] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 2 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:38.118809 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169899da82594d9cb673f8db74764ca2" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:38.119026 6839 raft_consensus.cc:3032] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:38.119102 6839 raft_consensus.cc:729] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:38.119341 6839 consensus_queue.cc:260] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.119586 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8fac59b25c164e658fe71cfdf7ca78cb"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:38.119697 6839 raft_consensus.cc:3037] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.119724 6577 raft_consensus.cc:465] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 21:27:38.120544 6579 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "093f8fd73ec34d8b9954c1716e910a78"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:38.120715 6579 raft_consensus.cc:465] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 21:27:38.121526 6579 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1325bd7bbb4340f88bdc7c705934862e"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:38.121718 6579 raft_consensus.cc:482] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.121793 6579 raft_consensus.cc:3037] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.122182 6839 raft_consensus.cc:2454] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 2.
I20240502 21:27:38.122709 8962 raft_consensus.cc:686] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:38.123080 8962 consensus_queue.cc:237] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.124426 6579 raft_consensus.cc:504] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.125648 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1325bd7bbb4340f88bdc7c705934862e" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.125810 6709 raft_consensus.cc:3032] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:38.125886 6709 raft_consensus.cc:729] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:38.126094 6709 consensus_queue.cc:260] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.126427 6709 raft_consensus.cc:3037] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.128754 6709 raft_consensus.cc:2454] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 2.
I20240502 21:27:38.129137 6515 leader_election.cc:302] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:38.129421 8962 raft_consensus.cc:2781] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:38.129580 8962 raft_consensus.cc:686] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:38.129797 8962 consensus_queue.cc:237] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.129917 6579 leader_election.cc:288] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 2 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:38.130026 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1325bd7bbb4340f88bdc7c705934862e" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.130264 6969 raft_consensus.cc:3037] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.130767 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "875f83053fee45b3bc1921b24745e9da"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:38.130950 6577 raft_consensus.cc:465] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 21:27:38.131891 6579 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "533d0913d0bc4582b976a58bda8176bc"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:38.132072 6579 raft_consensus.cc:482] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.132202 6579 raft_consensus.cc:3037] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.135869 6969 raft_consensus.cc:2454] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 2.
I20240502 21:27:38.137406 6579 raft_consensus.cc:504] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.138906 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533d0913d0bc4582b976a58bda8176bc" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.139078 6709 raft_consensus.cc:3032] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:38.139166 6709 raft_consensus.cc:729] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:38.139405 6709 consensus_queue.cc:260] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.139760 6709 raft_consensus.cc:3037] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.142995 6709 raft_consensus.cc:2454] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 2.
I20240502 21:27:38.143501 6515 leader_election.cc:302] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:38.144016 8962 raft_consensus.cc:2781] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:38.144248 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533d0913d0bc4582b976a58bda8176bc" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.144424 6969 raft_consensus.cc:3037] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.144691 8962 raft_consensus.cc:686] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:38.145239 8962 consensus_queue.cc:237] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.146512 6969 raft_consensus.cc:2454] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 2.
I20240502 21:27:38.146975 6579 leader_election.cc:288] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 2 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:38.148135 6579 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c9e728880af44303a975a63a4bd7bc62"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:38.148362 6579 raft_consensus.cc:482] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.148521 6579 raft_consensus.cc:3037] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.150840 6433 catalog_manager.cc:5484] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 1 to 2, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 2 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:38.151005 6579 raft_consensus.cc:504] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:38.151378 6433 catalog_manager.cc:5484] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 1 to 2, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 2 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.151836 6433 catalog_manager.cc:5484] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 1 to 2, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 2 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.152319 6577 raft_consensus.cc:1226] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Rejecting Update request from peer 80a838c00b544bf68ec4cc9c513212b2 for earlier term 1. Current term is 2. Ops: []
I20240502 21:27:38.152489 6579 leader_election.cc:288] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 2 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:38.152923 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e728880af44303a975a63a4bd7bc62" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.152951 8687 consensus_queue.cc:1046] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:38.152215 6433 catalog_manager.cc:5484] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 1 to 2, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 2 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.154230 6433 catalog_manager.cc:5484] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 1 to 2, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 2 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.155243 6433 catalog_manager.cc:5484] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 1 to 2, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 2 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.155853 6433 catalog_manager.cc:5484] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 1 to 2, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 2 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:38.156409 6433 catalog_manager.cc:5484] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 1 to 2, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 2 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.157032 6433 catalog_manager.cc:5484] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 1 to 2, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 2 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.157613 6433 catalog_manager.cc:5484] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 1 to 2, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 2 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:38.158289 6433 catalog_manager.cc:5484] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 1 to 2, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 2 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.158849 6433 catalog_manager.cc:5484] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 1 to 2, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 2 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.159441 6433 catalog_manager.cc:5484] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 1 to 2, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 2 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.161569 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e728880af44303a975a63a4bd7bc62" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.161783 6709 raft_consensus.cc:3037] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.167749 6709 raft_consensus.cc:2454] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 2.
I20240502 21:27:38.168403 6515 leader_election.cc:302] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:38.168812 8962 raft_consensus.cc:2781] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:38.168995 8962 raft_consensus.cc:686] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:38.169296 8962 consensus_queue.cc:237] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:38.172305 6970 raft_consensus.cc:3032] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:38.172469 6970 raft_consensus.cc:729] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.172775 6970 consensus_queue.cc:260] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:38.173277 6970 raft_consensus.cc:3037] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.176854 6970 raft_consensus.cc:2454] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 2.
I20240502 21:27:38.184137 6433 catalog_manager.cc:5484] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 1 to 2, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 2 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:38.201524 6970 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3093485f30b41939a1380ab7aa9e38b"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:38.201740 6970 raft_consensus.cc:482] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.201867 6970 raft_consensus.cc:3037] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.203559 6970 raft_consensus.cc:504] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.204188 6970 leader_election.cc:288] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 2 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:38.204582 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3093485f30b41939a1380ab7aa9e38b" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.204802 6709 raft_consensus.cc:3032] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:38.204767 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3093485f30b41939a1380ab7aa9e38b" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:38.204902 6709 raft_consensus.cc:729] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:38.204993 6839 raft_consensus.cc:3037] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.205296 6970 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b2b3e30b94d47dd822539cb30f36f81"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:38.205206 6709 consensus_queue.cc:260] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.205528 6970 raft_consensus.cc:482] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.205588 6709 raft_consensus.cc:3037] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.205621 6970 raft_consensus.cc:3037] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.207685 6839 raft_consensus.cc:2454] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 2.
I20240502 21:27:38.208154 6904 leader_election.cc:302] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:38.208192 6709 raft_consensus.cc:2454] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 2.
I20240502 21:27:38.208413 8988 raft_consensus.cc:2781] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:38.208618 8988 raft_consensus.cc:686] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 2 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.208920 8988 consensus_queue.cc:237] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.208981 6970 raft_consensus.cc:504] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.209857 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2b3e30b94d47dd822539cb30f36f81" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.210068 6970 leader_election.cc:288] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 2 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:38.210175 6709 raft_consensus.cc:3037] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.210350 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2b3e30b94d47dd822539cb30f36f81" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:38.210536 6839 raft_consensus.cc:3032] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:38.210660 6839 raft_consensus.cc:729] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:38.210884 6969 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aae0bc634aa7424e935c8bc81a85099b"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:38.211016 6969 raft_consensus.cc:482] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.211123 6969 raft_consensus.cc:3037] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.212476 6839 consensus_queue.cc:260] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.212810 6969 raft_consensus.cc:504] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.212869 6839 raft_consensus.cc:3037] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.213430 6969 leader_election.cc:288] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 3 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:38.213487 6709 raft_consensus.cc:2454] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 2.
I20240502 21:27:38.214447 6906 leader_election.cc:302] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:38.214632 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae0bc634aa7424e935c8bc81a85099b" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.214802 6709 raft_consensus.cc:3037] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.215021 8687 raft_consensus.cc:2781] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:38.215155 8687 raft_consensus.cc:686] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.215421 8687 consensus_queue.cc:237] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.217414 6839 raft_consensus.cc:2454] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 2.
I20240502 21:27:38.217433 6709 raft_consensus.cc:2454] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 3.
I20240502 21:27:38.218317 6906 leader_election.cc:302] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:38.218822 6969 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3619e8a307de4583a4e470ad583fd7bc"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:38.219041 6969 raft_consensus.cc:482] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.219188 6969 raft_consensus.cc:3037] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.222512 6969 raft_consensus.cc:504] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:38.223026 8687 raft_consensus.cc:2781] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:38.223232 8687 raft_consensus.cc:686] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 3 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.223557 8687 consensus_queue.cc:237] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.223770 6969 leader_election.cc:288] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 3 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:38.225464 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae0bc634aa7424e935c8bc81a85099b" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.225701 6577 raft_consensus.cc:3032] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:38.225808 6577 raft_consensus.cc:729] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:38.225909 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3619e8a307de4583a4e470ad583fd7bc" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.226053 6579 raft_consensus.cc:3032] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:38.226140 6579 raft_consensus.cc:729] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:38.226361 6579 consensus_queue.cc:260] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:38.226382 6577 consensus_queue.cc:260] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.226799 6577 raft_consensus.cc:3037] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.226917 6579 raft_consensus.cc:3037] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.229936 6579 raft_consensus.cc:2396] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.229983 6577 raft_consensus.cc:2396] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.230497 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3619e8a307de4583a4e470ad583fd7bc" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:38.230672 6839 raft_consensus.cc:3037] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.230891 6970 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3b6b526ec34c41aea64a2a19f0208f5d"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:38.231334 6970 raft_consensus.cc:482] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.231501 6970 raft_consensus.cc:3037] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.233475 6839 raft_consensus.cc:2454] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 3.
I20240502 21:27:38.233837 6904 leader_election.cc:302] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters: 51bc1a0939fe49348a18d3da18f9feae
I20240502 21:27:38.234153 6970 raft_consensus.cc:504] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.234975 6970 leader_election.cc:288] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 2 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:38.235973 6969 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c10bca264dda4180bd19beb5767bf047"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:38.236264 6969 raft_consensus.cc:482] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.236472 6969 raft_consensus.cc:3037] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.240722 6969 raft_consensus.cc:504] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.241583 6969 leader_election.cc:288] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 2 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:38.242764 6969 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "66198d766e6e4ceb820bc92bcca67d85"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:38.242942 6969 raft_consensus.cc:482] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.243181 6969 raft_consensus.cc:3037] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.247617 6969 raft_consensus.cc:504] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:38.248633 6969 leader_election.cc:288] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 3 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:38.248559 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6b526ec34c41aea64a2a19f0208f5d" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.248785 6577 raft_consensus.cc:3032] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:38.248878 6577 raft_consensus.cc:729] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:38.249176 6577 consensus_queue.cc:260] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.249545 6577 raft_consensus.cc:3037] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.249791 6970 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "331bcde3e6bf43c081af836c8cdebdc7"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:38.249961 6970 raft_consensus.cc:482] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.250083 6970 raft_consensus.cc:3037] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.252007 6577 raft_consensus.cc:2454] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 2.
I20240502 21:27:38.252758 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6b526ec34c41aea64a2a19f0208f5d" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:38.252751 6970 raft_consensus.cc:504] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.253006 6839 raft_consensus.cc:3037] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.253487 6970 leader_election.cc:288] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 2 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:38.248781 8687 raft_consensus.cc:2781] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:38.254395 6971 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57e37f7e15b74819b692792eaed94f24"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:38.254621 6971 raft_consensus.cc:465] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 21:27:38.255438 6839 raft_consensus.cc:2454] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 2.
I20240502 21:27:38.254422 8687 raft_consensus.cc:686] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 3 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.255745 8687 consensus_queue.cc:237] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:38.255901 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10bca264dda4180bd19beb5767bf047" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.256107 6709 raft_consensus.cc:3037] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.256342 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66198d766e6e4ceb820bc92bcca67d85" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:38.256392 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10bca264dda4180bd19beb5767bf047" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:38.256541 6840 raft_consensus.cc:3037] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.256621 6839 raft_consensus.cc:3032] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:38.256717 6839 raft_consensus.cc:729] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:38.256927 6839 consensus_queue.cc:260] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.257284 6839 raft_consensus.cc:3037] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.258819 6709 raft_consensus.cc:2454] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 2.
I20240502 21:27:38.259284 6840 raft_consensus.cc:2454] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 3.
I20240502 21:27:38.259322 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331bcde3e6bf43c081af836c8cdebdc7" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.259475 6711 raft_consensus.cc:3037] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.259954 6839 raft_consensus.cc:2454] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 2.
I20240502 21:27:38.260572 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66198d766e6e4ceb820bc92bcca67d85" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.260778 6577 raft_consensus.cc:3032] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:38.260891 6577 raft_consensus.cc:729] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:38.261152 6577 consensus_queue.cc:260] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:38.261390 6577 raft_consensus.cc:3037] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.261904 6711 raft_consensus.cc:2454] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 2.
I20240502 21:27:38.263325 6577 raft_consensus.cc:2396] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.263571 6969 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fbec2b1b80644a93b9239600157ad957"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:38.263747 6969 raft_consensus.cc:482] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.263830 6969 raft_consensus.cc:3037] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.264885 6904 leader_election.cc:302] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:38.265162 6904 leader_election.cc:302] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:38.265203 8992 raft_consensus.cc:2781] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:38.265465 6904 leader_election.cc:302] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:38.265462 8992 raft_consensus.cc:686] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 2 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.265899 8994 raft_consensus.cc:2781] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:38.266044 8994 raft_consensus.cc:686] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.265986 6969 raft_consensus.cc:504] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.265862 8992 consensus_queue.cc:237] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.266322 8994 consensus_queue.cc:237] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.266561 6969 leader_election.cc:288] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 2 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:38.267486 6969 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ea3b52c673ae4b9f931059c4b0a456fa"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:38.267686 6969 raft_consensus.cc:482] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.267699 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec2b1b80644a93b9239600157ad957" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.267833 6969 raft_consensus.cc:3037] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.267922 6579 raft_consensus.cc:3032] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:38.268041 6579 raft_consensus.cc:729] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:38.268365 8996 raft_consensus.cc:2781] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:38.268548 6579 consensus_queue.cc:260] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.269017 6579 raft_consensus.cc:3037] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.269779 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331bcde3e6bf43c081af836c8cdebdc7" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.270360 6577 raft_consensus.cc:3032] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:38.270663 6577 raft_consensus.cc:729] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:38.271082 6577 consensus_queue.cc:260] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.271895 6577 raft_consensus.cc:3037] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.272127 6579 raft_consensus.cc:2454] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 2.
I20240502 21:27:38.272912 6906 leader_election.cc:302] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:38.273721 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec2b1b80644a93b9239600157ad957" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:38.273934 6839 raft_consensus.cc:3037] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.273897 6969 raft_consensus.cc:504] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.274538 6969 leader_election.cc:288] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 2 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:38.275336 6971 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fb023d72aa8d4248a64d4723e18a3cf6"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:38.275508 6971 raft_consensus.cc:482] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.275656 6971 raft_consensus.cc:3037] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.277225 6839 raft_consensus.cc:2454] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 2.
I20240502 21:27:38.278175 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3b52c673ae4b9f931059c4b0a456fa" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:38.278439 6839 raft_consensus.cc:3032] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:38.278565 6839 raft_consensus.cc:729] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:38.278545 6971 raft_consensus.cc:504] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.278975 6839 consensus_queue.cc:260] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.279198 6971 leader_election.cc:288] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 3 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:38.279368 6839 raft_consensus.cc:3037] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.279934 6904 leader_election.cc:302] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:38.280042 6970 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "980f9cc0cdc6452d8009e9f75a0d4275"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:38.280215 6970 raft_consensus.cc:482] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.279178 6577 raft_consensus.cc:2454] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 2.
I20240502 21:27:38.280351 6970 raft_consensus.cc:3037] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.281256 8988 raft_consensus.cc:2781] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:38.281430 8988 raft_consensus.cc:686] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.281725 8988 consensus_queue.cc:237] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.282964 6970 raft_consensus.cc:504] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.283622 6970 leader_election.cc:288] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 2 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:38.284523 6970 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5e38d544ee14490c9fb5d8069440fd6a"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:38.284739 6970 raft_consensus.cc:482] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.284844 6970 raft_consensus.cc:3037] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.285977 6839 raft_consensus.cc:2454] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 2.
I20240502 21:27:38.287528 6970 raft_consensus.cc:504] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.288282 6970 leader_election.cc:288] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 3 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:38.289171 6971 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a09d2491d9dc4cbfb20f7adc1087b375"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:38.289368 6971 raft_consensus.cc:482] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.289490 6971 raft_consensus.cc:3037] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.291105 8988 raft_consensus.cc:2781] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:38.291257 8988 raft_consensus.cc:686] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.291533 8988 consensus_queue.cc:237] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.292326 6971 raft_consensus.cc:504] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.293025 6971 leader_election.cc:288] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 2 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:38.293942 6970 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5b2e443bad344ce89ce69871406c5a29"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:38.294103 6970 raft_consensus.cc:482] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.294224 6970 raft_consensus.cc:3037] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.294268 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb023d72aa8d4248a64d4723e18a3cf6" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.294521 6577 raft_consensus.cc:3032] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:38.294634 6577 raft_consensus.cc:729] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:38.294656 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e38d544ee14490c9fb5d8069440fd6a" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:38.294839 6840 raft_consensus.cc:3037] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.295014 6577 consensus_queue.cc:260] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.295326 6577 raft_consensus.cc:3037] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.297293 6970 raft_consensus.cc:504] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.297936 6970 leader_election.cc:288] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 3 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:38.297945 6840 raft_consensus.cc:2454] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 3.
I20240502 21:27:38.298163 6577 raft_consensus.cc:2396] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.298823 6971 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "093f8fd73ec34d8b9954c1716e910a78"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:38.299042 6971 raft_consensus.cc:482] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.299182 6971 raft_consensus.cc:3037] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.299412 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e38d544ee14490c9fb5d8069440fd6a" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.299603 6577 raft_consensus.cc:3032] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:38.299695 6577 raft_consensus.cc:729] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:38.299701 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09d2491d9dc4cbfb20f7adc1087b375" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:38.299932 6840 raft_consensus.cc:3032] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:38.300077 6840 raft_consensus.cc:729] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:38.299984 6577 consensus_queue.cc:260] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.300313 6577 raft_consensus.cc:3037] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.300529 6840 consensus_queue.cc:260] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.300985 6840 raft_consensus.cc:3037] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.301649 6971 raft_consensus.cc:504] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.302939 6971 leader_election.cc:288] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 2 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:38.303892 8996 raft_consensus.cc:686] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.303912 6840 raft_consensus.cc:2454] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 2.
I20240502 21:27:38.304278 8996 consensus_queue.cc:237] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:38.306262 6971 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1325bd7bbb4340f88bdc7c705934862e"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:38.306329 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3b52c673ae4b9f931059c4b0a456fa" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.306483 6971 raft_consensus.cc:482] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.306602 6971 raft_consensus.cc:3037] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.306586 6711 raft_consensus.cc:3037] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.309337 6971 raft_consensus.cc:504] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.309935 6971 leader_election.cc:288] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 3 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:38.310024 6711 raft_consensus.cc:2454] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 2.
I20240502 21:27:38.310832 6969 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "875f83053fee45b3bc1921b24745e9da"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:38.310887 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb023d72aa8d4248a64d4723e18a3cf6" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.311113 6711 raft_consensus.cc:3037] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.311127 6969 raft_consensus.cc:482] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.311348 6969 raft_consensus.cc:3037] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.312745 6970 raft_consensus.cc:1226] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Rejecting Update request from peer 51bc1a0939fe49348a18d3da18f9feae for earlier term 1. Current term is 2. Ops: []
I20240502 21:27:38.312827 6577 raft_consensus.cc:2396] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.313345 8962 consensus_queue.cc:1046] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:38.313649 8962 raft_consensus.cc:3032] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:38.313747 8962 raft_consensus.cc:729] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:38.313990 8962 consensus_queue.cc:260] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.314322 6711 raft_consensus.cc:2454] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 3.
I20240502 21:27:38.314368 8962 raft_consensus.cc:3037] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.314981 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "980f9cc0cdc6452d8009e9f75a0d4275" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.315302 6711 raft_consensus.cc:3037] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.318307 6711 raft_consensus.cc:2454] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 2.
I20240502 21:27:38.319833 6710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1325bd7bbb4340f88bdc7c705934862e" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.320029 6710 raft_consensus.cc:3037] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.323020 6710 raft_consensus.cc:2454] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 3.
I20240502 21:27:38.323863 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "980f9cc0cdc6452d8009e9f75a0d4275" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:38.305145 6904 leader_election.cc:302] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:38.324795 8992 raft_consensus.cc:2781] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:38.325117 6839 raft_consensus.cc:3032] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:38.325281 8992 raft_consensus.cc:686] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 2 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.325570 8992 consensus_queue.cc:237] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.329030 6839 raft_consensus.cc:729] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:38.329383 6839 consensus_queue.cc:260] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.329783 6839 raft_consensus.cc:3037] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.331914 6906 leader_election.cc:302] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:38.332571 6839 raft_consensus.cc:2454] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 2.
I20240502 21:27:38.333475 6906 leader_election.cc:302] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:38.334331 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09d2491d9dc4cbfb20f7adc1087b375" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.334501 6711 raft_consensus.cc:3037] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.333743 8998 raft_consensus.cc:2781] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:38.333087 6904 leader_election.cc:302] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:38.335106 8992 raft_consensus.cc:2781] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:38.335266 8992 raft_consensus.cc:686] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.335428 8996 raft_consensus.cc:2781] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:38.335587 8992 consensus_queue.cc:237] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.335896 6904 leader_election.cc:302] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:38.336625 8998 raft_consensus.cc:686] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.336628 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2e443bad344ce89ce69871406c5a29" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.336813 6577 raft_consensus.cc:3032] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:38.336866 6711 raft_consensus.cc:2454] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 2.
I20240502 21:27:38.336908 6577 raft_consensus.cc:729] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:38.337164 6577 consensus_queue.cc:260] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.337136 8998 consensus_queue.cc:237] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.337460 6577 raft_consensus.cc:3037] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.338286 8988 raft_consensus.cc:2781] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:38.339718 8988 raft_consensus.cc:686] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.340157 6577 raft_consensus.cc:2396] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.340032 8988 consensus_queue.cc:237] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.335688 8996 raft_consensus.cc:686] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 3 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.340828 8996 consensus_queue.cc:237] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.342335 6906 leader_election.cc:302] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:38.342833 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093f8fd73ec34d8b9954c1716e910a78" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.343955 8687 raft_consensus.cc:2781] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:38.344190 8687 raft_consensus.cc:686] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 3 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.344568 8687 consensus_queue.cc:237] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.345789 6577 raft_consensus.cc:2454] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 2.
I20240502 21:27:38.346186 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2e443bad344ce89ce69871406c5a29" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:38.346387 6839 raft_consensus.cc:3037] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.348989 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093f8fd73ec34d8b9954c1716e910a78" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:38.349236 6840 raft_consensus.cc:3037] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.349880 6969 raft_consensus.cc:504] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.350728 6969 leader_election.cc:288] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 2 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:38.351740 6970 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "444ccd95dc0f4bdd806ca9151d56d5cd"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:38.351924 6970 raft_consensus.cc:465] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 21:27:38.351958 6840 raft_consensus.cc:2454] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 2.
I20240502 21:27:38.352035 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875f83053fee45b3bc1921b24745e9da" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.352291 6711 raft_consensus.cc:3037] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.352772 6971 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c9e728880af44303a975a63a4bd7bc62"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:38.352799 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1325bd7bbb4340f88bdc7c705934862e" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.352952 6971 raft_consensus.cc:482] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.352972 6577 raft_consensus.cc:3032] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:38.353077 6971 raft_consensus.cc:3037] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.353089 6577 raft_consensus.cc:729] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:38.353346 6577 consensus_queue.cc:260] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.353578 6577 raft_consensus.cc:3037] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.355412 6711 raft_consensus.cc:2454] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 2.
I20240502 21:27:38.355603 6971 raft_consensus.cc:504] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:38.356081 6577 raft_consensus.cc:2396] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.356165 6906 leader_election.cc:302] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:38.356283 6971 leader_election.cc:288] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 3 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:38.356947 6904 leader_election.cc:302] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2; no voters:
I20240502 21:27:38.356922 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e728880af44303a975a63a4bd7bc62" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.357133 6711 raft_consensus.cc:3037] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.357118 6968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "533d0913d0bc4582b976a58bda8176bc"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:38.357261 6968 raft_consensus.cc:482] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.357416 6968 raft_consensus.cc:3037] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.357836 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875f83053fee45b3bc1921b24745e9da" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.358012 6579 raft_consensus.cc:3032] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:38.358101 6579 raft_consensus.cc:729] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:38.358341 6579 consensus_queue.cc:260] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.358697 6579 raft_consensus.cc:3037] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.360224 6968 raft_consensus.cc:504] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.360663 6711 raft_consensus.cc:2454] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 3.
I20240502 21:27:38.361272 6579 raft_consensus.cc:2454] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 2.
I20240502 21:27:38.361333 6968 leader_election.cc:288] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 3 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:38.361672 9003 raft_consensus.cc:2781] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:38.361822 9003 raft_consensus.cc:686] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.361860 6906 leader_election.cc:302] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:38.361917 9002 raft_consensus.cc:2781] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:38.362144 9003 consensus_queue.cc:237] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.362371 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533d0913d0bc4582b976a58bda8176bc" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.362602 6711 raft_consensus.cc:3037] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.363235 9001 raft_consensus.cc:2781] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:38.363369 9001 raft_consensus.cc:686] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.363651 9001 consensus_queue.cc:237] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:38.372357 6711 raft_consensus.cc:2454] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 3.
I20240502 21:27:38.372892 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e728880af44303a975a63a4bd7bc62" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.373099 6577 raft_consensus.cc:3032] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:38.373190 6577 raft_consensus.cc:729] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:38.373466 6577 consensus_queue.cc:260] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:38.373762 6577 raft_consensus.cc:3037] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.377485 9002 raft_consensus.cc:686] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 2 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.377877 6839 raft_consensus.cc:2454] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 3.
I20240502 21:27:38.377859 9002 consensus_queue.cc:237] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.385241 6906 leader_election.cc:302] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:38.385493 8994 raft_consensus.cc:2781] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:38.376420 6577 raft_consensus.cc:2396] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.385828 8994 raft_consensus.cc:686] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 3 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.386109 8994 consensus_queue.cc:237] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.386358 6904 leader_election.cc:302] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters: 51bc1a0939fe49348a18d3da18f9feae
I20240502 21:27:38.387172 9002 raft_consensus.cc:2781] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:38.387356 9002 raft_consensus.cc:686] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.387588 9002 consensus_queue.cc:237] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.387744 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533d0913d0bc4582b976a58bda8176bc" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.387950 6577 raft_consensus.cc:3032] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:38.388046 6577 raft_consensus.cc:729] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:38.388329 6577 consensus_queue.cc:260] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.388695 6577 raft_consensus.cc:3037] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.391352 6577 raft_consensus.cc:2396] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.399695 6433 catalog_manager.cc:5484] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 1 to 2, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 2 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.400379 6433 catalog_manager.cc:5484] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 2 to 3, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 3 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:38.401468 6433 catalog_manager.cc:5484] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 2 to 3, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 3 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:38.402158 6433 catalog_manager.cc:5484] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 1 to 2, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 2 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.403007 6433 catalog_manager.cc:5484] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 1 to 2, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 2 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.403575 6433 catalog_manager.cc:5484] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 1 to 2, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 2 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.404098 6433 catalog_manager.cc:5484] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 1 to 2, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 2 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.404683 6433 catalog_manager.cc:5484] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 2 to 3, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 3 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.405308 6433 catalog_manager.cc:5484] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 2 to 3, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 3 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:38.405855 6433 catalog_manager.cc:5484] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 2 to 3, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 3 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.406402 6433 catalog_manager.cc:5484] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 1 to 2, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 2 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:38.406909 6433 catalog_manager.cc:5484] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 1 to 2, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 2 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:38.407455 6433 catalog_manager.cc:5484] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 2 to 3, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 3 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:38.408012 6433 catalog_manager.cc:5484] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 2 to 3, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 3 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:38.408648 6433 catalog_manager.cc:5484] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 1 to 2, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 2 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.409109 6433 catalog_manager.cc:5484] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 1 to 2, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 2 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.409610 6433 catalog_manager.cc:5484] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 2 to 3, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 3 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.410140 6433 catalog_manager.cc:5484] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 1 to 2, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 2 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.410766 6433 catalog_manager.cc:5484] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 2 to 3, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 3 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:38.411285 6433 catalog_manager.cc:5484] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 1 to 2, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 2 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.417819 6711 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b2b3e30b94d47dd822539cb30f36f81"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:38.418073 6711 raft_consensus.cc:482] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.418210 6711 raft_consensus.cc:3037] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.420825 6839 raft_consensus.cc:1261] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Refusing update from remote peer 51bc1a0939fe49348a18d3da18f9feae: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:27:38.421025 6711 raft_consensus.cc:504] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.421504 8962 consensus_queue.cc:1035] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:38.422091 6711 leader_election.cc:288] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 3 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:38.422276 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2b3e30b94d47dd822539cb30f36f81" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.422480 6969 raft_consensus.cc:3032] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:38.422569 6969 raft_consensus.cc:729] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.422806 6969 consensus_queue.cc:260] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.423074 6969 raft_consensus.cc:3037] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.423499 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2b3e30b94d47dd822539cb30f36f81" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:38.423703 6840 raft_consensus.cc:3037] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.424299 6711 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "df917e7196ed4939b75dc52930c87640"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:38.424526 6711 raft_consensus.cc:482] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.424656 6711 raft_consensus.cc:3037] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.425688 6969 raft_consensus.cc:2396] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.426383 6840 raft_consensus.cc:2454] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 3.
I20240502 21:27:38.427268 6711 raft_consensus.cc:504] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.428179 6711 leader_election.cc:288] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 3 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:38.428999 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df917e7196ed4939b75dc52930c87640" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.429256 6577 raft_consensus.cc:3032] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:38.429352 6577 raft_consensus.cc:729] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:38.429313 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df917e7196ed4939b75dc52930c87640" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:38.429665 6577 consensus_queue.cc:260] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.429966 6577 raft_consensus.cc:3037] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.430358 6644 leader_election.cc:302] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:38.430673 8274 raft_consensus.cc:2781] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:38.433629 6577 raft_consensus.cc:2396] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.437742 6644 leader_election.cc:302] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720
I20240502 21:27:38.438591 9027 raft_consensus.cc:2726] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 21:27:38.439124 6711 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aae0bc634aa7424e935c8bc81a85099b"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:38.439321 6711 raft_consensus.cc:482] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.439452 6711 raft_consensus.cc:3037] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.442492 6839 raft_consensus.cc:1261] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Refusing update from remote peer 51bc1a0939fe49348a18d3da18f9feae: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:27:38.442495 6711 raft_consensus.cc:504] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.443178 8962 consensus_queue.cc:1035] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:38.443636 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae0bc634aa7424e935c8bc81a85099b" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.443848 6969 raft_consensus.cc:3032] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:38.443962 6969 raft_consensus.cc:729] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.444216 6969 consensus_queue.cc:260] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.444579 6969 raft_consensus.cc:3037] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.445175 8274 raft_consensus.cc:686] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:38.445267 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae0bc634aa7424e935c8bc81a85099b" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.445505 6577 raft_consensus.cc:3037] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.445528 8274 consensus_queue.cc:237] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.447094 6969 raft_consensus.cc:2396] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.447526 6711 leader_election.cc:288] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 4 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:38.448395 6711 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:38.448607 6711 raft_consensus.cc:482] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.448695 6711 raft_consensus.cc:3037] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.450457 6577 raft_consensus.cc:2396] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.450961 6644 leader_election.cc:302] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:38.451606 6711 raft_consensus.cc:504] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.452298 6711 leader_election.cc:288] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 3 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:38.452634 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.452803 6577 raft_consensus.cc:3032] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:38.452888 6577 raft_consensus.cc:729] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:38.452936 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:38.453112 6839 raft_consensus.cc:3037] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.453215 6577 consensus_queue.cc:260] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.453475 6577 raft_consensus.cc:3037] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.455222 6839 raft_consensus.cc:2454] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 3.
I20240502 21:27:38.456255 6644 leader_election.cc:302] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:38.456504 9027 raft_consensus.cc:2726] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:27:38.456748 9027 raft_consensus.cc:2781] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:38.456915 9027 raft_consensus.cc:686] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:38.457036 6577 raft_consensus.cc:2396] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.457288 9027 consensus_queue.cc:237] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.460636 6711 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c10bca264dda4180bd19beb5767bf047"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:38.460817 6711 raft_consensus.cc:482] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.460943 6711 raft_consensus.cc:3037] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.463819 6711 raft_consensus.cc:504] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.465224 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10bca264dda4180bd19beb5767bf047" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.465456 6969 raft_consensus.cc:3032] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:38.465564 6969 raft_consensus.cc:729] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.465833 6969 consensus_queue.cc:260] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.466068 6969 raft_consensus.cc:3037] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.468405 6969 raft_consensus.cc:2396] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.469754 6711 leader_election.cc:288] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 3 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:38.470229 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10bca264dda4180bd19beb5767bf047" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:38.470458 6839 raft_consensus.cc:3037] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.472723 6839 raft_consensus.cc:2454] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 3.
I20240502 21:27:38.473510 6644 leader_election.cc:302] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:38.473994 8274 raft_consensus.cc:2781] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:38.474113 8274 raft_consensus.cc:686] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:38.474305 8274 consensus_queue.cc:237] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.475623 6433 catalog_manager.cc:5484] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 2 to 3, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 3 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.481074 6710 raft_consensus.cc:1261] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Refusing update from remote peer 51bc1a0939fe49348a18d3da18f9feae: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:27:38.481274 6711 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "da14dcc58d684704b3649e85b26fb43c"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:38.481424 6711 raft_consensus.cc:482] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.481509 6711 raft_consensus.cc:3037] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.481595 6433 catalog_manager.cc:5484] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 2 to 3, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 3 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.482131 6433 catalog_manager.cc:5484] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 2 to 3, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 3 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:38.483716 6711 raft_consensus.cc:504] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:38.484014 8965 consensus_queue.cc:1035] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:38.484629 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da14dcc58d684704b3649e85b26fb43c" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:38.484788 6839 raft_consensus.cc:3037] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.485432 8815 heartbeater.cc:503] Master 127.24.226.190:40767 was elected leader, sending a full tablet report...
I20240502 21:27:38.487049 6839 raft_consensus.cc:2454] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 3.
I20240502 21:27:38.487566 6644 leader_election.cc:302] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:38.487906 9025 raft_consensus.cc:2781] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:38.488114 9025 raft_consensus.cc:686] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:38.488596 9025 consensus_queue.cc:237] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:38.493468 6711 leader_election.cc:288] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 3 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:38.493947 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da14dcc58d684704b3649e85b26fb43c" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.494122 6577 raft_consensus.cc:3032] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:38.494202 6577 raft_consensus.cc:729] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:38.494411 6577 consensus_queue.cc:260] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:38.494652 6577 raft_consensus.cc:3037] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.494850 6711 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f0b941baaab542a78748d4d0894cfbf4"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:38.496850 6577 raft_consensus.cc:2396] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.500831 6711 raft_consensus.cc:482] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.500996 6711 raft_consensus.cc:3037] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.502902 6711 raft_consensus.cc:504] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.503870 6711 leader_election.cc:288] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 3 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:38.504417 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b941baaab542a78748d4d0894cfbf4" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:38.504482 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b941baaab542a78748d4d0894cfbf4" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.504638 6839 raft_consensus.cc:3037] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.504899 6577 raft_consensus.cc:3032] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:38.505012 6577 raft_consensus.cc:729] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:38.505373 6577 consensus_queue.cc:260] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.505723 6577 raft_consensus.cc:3037] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.506968 6839 raft_consensus.cc:2454] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 3.
I20240502 21:27:38.507310 6644 leader_election.cc:302] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:38.507606 9025 raft_consensus.cc:2781] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:38.507773 9025 raft_consensus.cc:686] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:38.507953 6710 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fb023d72aa8d4248a64d4723e18a3cf6"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:38.508018 9025 consensus_queue.cc:237] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.508131 6710 raft_consensus.cc:482] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.508245 6710 raft_consensus.cc:3037] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.508695 6577 raft_consensus.cc:2454] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 3.
I20240502 21:27:38.510134 6433 catalog_manager.cc:5484] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 2 to 3, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 3 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.510957 6710 raft_consensus.cc:504] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.511958 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb023d72aa8d4248a64d4723e18a3cf6" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.512105 6577 raft_consensus.cc:3037] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.512360 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb023d72aa8d4248a64d4723e18a3cf6" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.512565 6710 leader_election.cc:288] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 4 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:38.512604 6969 raft_consensus.cc:3032] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:38.512692 6969 raft_consensus.cc:729] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.512938 6969 consensus_queue.cc:260] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.513255 6969 raft_consensus.cc:3037] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.513389 6710 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "980f9cc0cdc6452d8009e9f75a0d4275"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:38.513543 6710 raft_consensus.cc:482] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.513646 6710 raft_consensus.cc:3037] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.514456 6577 raft_consensus.cc:2396] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.516176 6969 raft_consensus.cc:2396] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.516122 6710 raft_consensus.cc:504] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.516700 6645 leader_election.cc:302] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:38.516722 6710 leader_election.cc:288] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 3 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:38.517575 9025 raft_consensus.cc:2726] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:27:38.518357 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "980f9cc0cdc6452d8009e9f75a0d4275" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:38.518443 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "980f9cc0cdc6452d8009e9f75a0d4275" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.518596 6839 raft_consensus.cc:3037] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.518810 6969 raft_consensus.cc:3032] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:38.518946 6969 raft_consensus.cc:729] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.519208 6969 consensus_queue.cc:260] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.519464 6969 raft_consensus.cc:3037] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.520047 6433 catalog_manager.cc:5484] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 2 to 3, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 3 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:38.520892 6839 raft_consensus.cc:2454] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 3.
I20240502 21:27:38.521309 6969 raft_consensus.cc:2396] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.522137 6644 leader_election.cc:302] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:38.522246 6710 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57e37f7e15b74819b692792eaed94f24"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:38.522414 6710 raft_consensus.cc:482] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.522534 6710 raft_consensus.cc:3037] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.522647 9025 raft_consensus.cc:2781] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:38.522809 9025 raft_consensus.cc:686] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:38.522969 9025 consensus_queue.cc:237] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.525261 6710 raft_consensus.cc:504] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.525729 6710 leader_election.cc:288] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 2 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:38.526094 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e37f7e15b74819b692792eaed94f24" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:38.526275 6839 raft_consensus.cc:3037] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.526717 6710 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "331bcde3e6bf43c081af836c8cdebdc7"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:38.526901 6710 raft_consensus.cc:482] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.527000 6710 raft_consensus.cc:3037] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.528087 6839 raft_consensus.cc:2454] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 2.
I20240502 21:27:38.528419 6644 leader_election.cc:302] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:38.529577 6710 raft_consensus.cc:504] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.530284 6710 leader_election.cc:288] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 3 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:38.530386 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331bcde3e6bf43c081af836c8cdebdc7" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.530558 6970 raft_consensus.cc:3032] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:38.530648 6970 raft_consensus.cc:729] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.530848 6970 consensus_queue.cc:260] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.531077 6970 raft_consensus.cc:3037] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.532545 9025 raft_consensus.cc:2781] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:38.532675 9025 raft_consensus.cc:686] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:38.532763 6970 raft_consensus.cc:2396] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.532910 9025 consensus_queue.cc:237] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.533385 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331bcde3e6bf43c081af836c8cdebdc7" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.533579 6577 raft_consensus.cc:3037] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.536016 6577 raft_consensus.cc:2454] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 3.
I20240502 21:27:38.536351 6644 leader_election.cc:302] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:38.536607 9025 raft_consensus.cc:2781] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:38.536388 6433 catalog_manager.cc:5484] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 2 to 3, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 3 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:38.536739 9025 raft_consensus.cc:686] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:38.536948 9025 consensus_queue.cc:237] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.537472 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e37f7e15b74819b692792eaed94f24" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.537657 6969 raft_consensus.cc:3032] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:38.537771 6969 raft_consensus.cc:729] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.538163 6969 consensus_queue.cc:260] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.538532 6969 raft_consensus.cc:3037] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.540589 6710 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c6d11cba6ca04c17a9f47506824d9aa2"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:38.540792 6710 raft_consensus.cc:482] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.541074 6969 raft_consensus.cc:2454] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 2.
I20240502 21:27:38.540978 6710 raft_consensus.cc:3037] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.541098 6433 catalog_manager.cc:5484] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 1 to 2, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 2 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.541818 6433 catalog_manager.cc:5484] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 2 to 3, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 3 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:38.543398 6710 raft_consensus.cc:504] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.544016 6710 leader_election.cc:288] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 3 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:38.544705 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d11cba6ca04c17a9f47506824d9aa2" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:38.544734 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d11cba6ca04c17a9f47506824d9aa2" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.544875 6839 raft_consensus.cc:3037] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.544898 6577 raft_consensus.cc:3032] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:38.544987 6577 raft_consensus.cc:729] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:38.545233 6577 consensus_queue.cc:260] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.545513 6577 raft_consensus.cc:3037] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.547122 6839 raft_consensus.cc:2454] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 3.
I20240502 21:27:38.547477 6644 leader_election.cc:302] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:38.547688 6710 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "444ccd95dc0f4bdd806ca9151d56d5cd"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:38.547797 9025 raft_consensus.cc:2781] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:38.547969 9025 raft_consensus.cc:686] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:38.547979 6710 raft_consensus.cc:482] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.548210 6710 raft_consensus.cc:3037] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.548341 6577 raft_consensus.cc:2396] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.548271 9025 consensus_queue.cc:237] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.550702 6710 raft_consensus.cc:504] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.551306 6710 leader_election.cc:288] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 2 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:38.551771 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444ccd95dc0f4bdd806ca9151d56d5cd" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.551980 6969 raft_consensus.cc:3032] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:38.552042 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444ccd95dc0f4bdd806ca9151d56d5cd" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:38.552102 6969 raft_consensus.cc:729] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.552232 6839 raft_consensus.cc:3037] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.552312 6710 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3093485f30b41939a1380ab7aa9e38b"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:38.552488 6710 raft_consensus.cc:482] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.552604 6710 raft_consensus.cc:3037] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.552531 6969 consensus_queue.cc:260] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.553062 6969 raft_consensus.cc:3037] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.554625 6433 catalog_manager.cc:5484] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 2 to 3, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 3 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:38.554955 6710 raft_consensus.cc:504] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.555706 6969 raft_consensus.cc:2454] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 2.
I20240502 21:27:38.555898 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3093485f30b41939a1380ab7aa9e38b" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.556102 6970 raft_consensus.cc:3032] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:38.556059 6839 raft_consensus.cc:2454] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 2.
I20240502 21:27:38.556197 6970 raft_consensus.cc:729] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.556421 6970 consensus_queue.cc:260] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.556569 6645 leader_election.cc:302] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:38.556717 6970 raft_consensus.cc:3037] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.556787 9025 raft_consensus.cc:2781] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:38.556934 9025 raft_consensus.cc:686] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:38.557143 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3093485f30b41939a1380ab7aa9e38b" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:38.557150 9025 consensus_queue.cc:237] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.557305 6839 raft_consensus.cc:3037] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.559270 6970 raft_consensus.cc:2396] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.559787 6839 raft_consensus.cc:2454] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 3.
I20240502 21:27:38.560187 6710 leader_election.cc:288] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 3 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:38.560163 6644 leader_election.cc:302] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:38.560559 9025 raft_consensus.cc:2781] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:38.560700 9025 raft_consensus.cc:686] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:38.560930 9025 consensus_queue.cc:237] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.561626 6433 catalog_manager.cc:5484] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 1 to 2, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 2 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.562032 6710 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1c98cbfbd0a44335b75b228c217f9132"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:38.562224 6710 raft_consensus.cc:482] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.562345 6710 raft_consensus.cc:3037] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.563915 6710 raft_consensus.cc:504] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:38.564397 6710 leader_election.cc:288] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 2 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:38.564946 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c98cbfbd0a44335b75b228c217f9132" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.565130 6577 raft_consensus.cc:3032] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:38.565222 6577 raft_consensus.cc:729] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:38.565466 6577 consensus_queue.cc:260] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:38.565694 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c98cbfbd0a44335b75b228c217f9132" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:38.565805 6577 raft_consensus.cc:3037] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.565877 6839 raft_consensus.cc:3037] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.566573 6433 catalog_manager.cc:5484] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 2 to 3, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 3 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.568213 6839 raft_consensus.cc:2454] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 2.
I20240502 21:27:38.568575 6577 raft_consensus.cc:2454] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 2.
I20240502 21:27:38.568662 6644 leader_election.cc:302] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:38.568920 6710 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ea3b52c673ae4b9f931059c4b0a456fa"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:38.569100 6710 raft_consensus.cc:482] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.569239 6710 raft_consensus.cc:3037] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.569284 9025 raft_consensus.cc:2781] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:38.569442 9025 raft_consensus.cc:686] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:38.569681 9025 consensus_queue.cc:237] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:38.571521 6710 raft_consensus.cc:504] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.571996 6710 leader_election.cc:288] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 3 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:38.572397 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3b52c673ae4b9f931059c4b0a456fa" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:38.572592 6839 raft_consensus.cc:3037] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.572520 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3b52c673ae4b9f931059c4b0a456fa" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.572782 6970 raft_consensus.cc:3032] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:38.572911 6970 raft_consensus.cc:729] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.573206 6970 consensus_queue.cc:260] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.573529 6970 raft_consensus.cc:3037] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.574247 6839 raft_consensus.cc:2454] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 3.
I20240502 21:27:38.574470 6710 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "169899da82594d9cb673f8db74764ca2"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:38.574613 6644 leader_election.cc:302] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:38.574636 6710 raft_consensus.cc:482] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.574784 6710 raft_consensus.cc:3037] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.574916 9025 raft_consensus.cc:2781] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:38.575230 9025 raft_consensus.cc:686] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:38.575507 9025 consensus_queue.cc:237] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.576282 6970 raft_consensus.cc:2396] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.577757 6710 raft_consensus.cc:504] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.578563 6710 leader_election.cc:288] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 3 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:38.579011 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169899da82594d9cb673f8db74764ca2" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.579210 6577 raft_consensus.cc:3032] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:38.579320 6577 raft_consensus.cc:729] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:38.579581 6577 consensus_queue.cc:260] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.579867 6577 raft_consensus.cc:3037] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.579823 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169899da82594d9cb673f8db74764ca2" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:38.580019 6839 raft_consensus.cc:3037] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.579901 6710 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a09d2491d9dc4cbfb20f7adc1087b375"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:38.580292 6710 raft_consensus.cc:482] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.580504 6710 raft_consensus.cc:3037] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.582408 6839 raft_consensus.cc:2454] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 3.
I20240502 21:27:38.582523 6577 raft_consensus.cc:2396] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.582782 6644 leader_election.cc:302] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:38.583030 8274 raft_consensus.cc:2781] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:38.583205 8274 raft_consensus.cc:686] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:38.583164 6710 raft_consensus.cc:504] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.583488 8274 consensus_queue.cc:237] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.583979 6710 leader_election.cc:288] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 3 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:38.584261 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09d2491d9dc4cbfb20f7adc1087b375" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.584527 6970 raft_consensus.cc:3032] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:38.584637 6970 raft_consensus.cc:729] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.584853 6970 consensus_queue.cc:260] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.585125 6970 raft_consensus.cc:3037] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.585433 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09d2491d9dc4cbfb20f7adc1087b375" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:38.585600 6840 raft_consensus.cc:3037] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.587725 6970 raft_consensus.cc:2396] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.588106 6840 raft_consensus.cc:2454] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 3.
I20240502 21:27:38.592865 6644 leader_election.cc:302] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:38.593462 6710 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8fac59b25c164e658fe71cfdf7ca78cb"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:38.593736 6710 raft_consensus.cc:482] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.593834 6710 raft_consensus.cc:3037] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.593886 6433 catalog_manager.cc:5484] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 1 to 2, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 2 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.594543 9025 raft_consensus.cc:2781] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:38.594370 6433 catalog_manager.cc:5484] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 2 to 3, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 3 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.594743 9025 raft_consensus.cc:686] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:38.595021 9025 consensus_queue.cc:237] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.598776 6710 raft_consensus.cc:504] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:38.599411 6710 leader_election.cc:288] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 2 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:38.600064 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fac59b25c164e658fe71cfdf7ca78cb" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.600246 6577 raft_consensus.cc:3032] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:38.600369 6577 raft_consensus.cc:729] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:38.600915 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fac59b25c164e658fe71cfdf7ca78cb" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:38.600903 6577 consensus_queue.cc:260] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:38.601172 6839 raft_consensus.cc:3037] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.601222 6710 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1325bd7bbb4340f88bdc7c705934862e"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:38.601332 6577 raft_consensus.cc:3037] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:38.601089 6433 catalog_manager.cc:5484] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 2 to 3, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 3 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.601745 6433 catalog_manager.cc:5484] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 2 to 3, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 3 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:38.603830 6577 raft_consensus.cc:2454] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 2.
I20240502 21:27:38.604229 6839 raft_consensus.cc:2454] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 2.
I20240502 21:27:38.604184 6644 leader_election.cc:302] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:38.606289 6710 raft_consensus.cc:482] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.606524 9025 raft_consensus.cc:2781] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:38.606613 6710 raft_consensus.cc:3037] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.606721 9025 raft_consensus.cc:686] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:38.606953 9025 consensus_queue.cc:237] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:38.609067 6710 raft_consensus.cc:504] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.609638 6710 leader_election.cc:288] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 4 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:38.610445 6710 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "875f83053fee45b3bc1921b24745e9da"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:38.610625 6710 raft_consensus.cc:482] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.610721 6710 raft_consensus.cc:3037] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.610891 6433 catalog_manager.cc:5484] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 1 to 2, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 2 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.611617 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1325bd7bbb4340f88bdc7c705934862e" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.611745 6577 raft_consensus.cc:3037] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.612746 6710 raft_consensus.cc:504] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.613337 6710 leader_election.cc:288] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 3 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:38.613898 6577 raft_consensus.cc:2396] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.614171 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875f83053fee45b3bc1921b24745e9da" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.614208 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1325bd7bbb4340f88bdc7c705934862e" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.614328 6579 raft_consensus.cc:3037] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.614411 6970 raft_consensus.cc:3032] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:38.614498 6970 raft_consensus.cc:729] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.614647 6710 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "533d0913d0bc4582b976a58bda8176bc"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:38.614758 6710 raft_consensus.cc:482] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.614830 6710 raft_consensus.cc:3037] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.614722 6970 consensus_queue.cc:260] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.614998 6970 raft_consensus.cc:3037] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.616328 6710 raft_consensus.cc:504] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.616813 6579 raft_consensus.cc:2454] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 3.
I20240502 21:27:38.616842 6710 leader_election.cc:288] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 4 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:38.617377 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875f83053fee45b3bc1921b24745e9da" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.617543 6710 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c9e728880af44303a975a63a4bd7bc62"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:38.617584 6969 raft_consensus.cc:3032] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:38.617672 6710 raft_consensus.cc:482] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.617736 6710 raft_consensus.cc:3037] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.617691 6969 raft_consensus.cc:729] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.617738 6970 raft_consensus.cc:2396] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.617949 6969 consensus_queue.cc:260] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.618208 6969 raft_consensus.cc:3037] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.619493 6710 raft_consensus.cc:504] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:38.619989 6710 leader_election.cc:288] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 4 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:38.620828 6969 raft_consensus.cc:2396] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.621042 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533d0913d0bc4582b976a58bda8176bc" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.621214 6970 raft_consensus.cc:3032] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:38.621248 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e728880af44303a975a63a4bd7bc62" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.621273 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e728880af44303a975a63a4bd7bc62" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.621331 6970 raft_consensus.cc:729] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.621430 6577 raft_consensus.cc:3037] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.621444 6968 raft_consensus.cc:3032] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:38.621519 6968 raft_consensus.cc:729] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.621590 6970 consensus_queue.cc:260] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.621742 6968 consensus_queue.cc:260] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:38.621856 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533d0913d0bc4582b976a58bda8176bc" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.621924 6970 raft_consensus.cc:3037] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.621984 6968 raft_consensus.cc:3037] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.622035 6579 raft_consensus.cc:3037] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.624200 6577 raft_consensus.cc:2396] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.624702 6579 raft_consensus.cc:2396] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.624784 6968 raft_consensus.cc:2396] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.624811 6970 raft_consensus.cc:2396] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.625592 6644 leader_election.cc:302] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:38.625981 6644 leader_election.cc:302] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:38.626238 8274 raft_consensus.cc:2726] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:27:38.626372 6644 leader_election.cc:302] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:38.626395 8274 raft_consensus.cc:2781] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:38.626590 8274 raft_consensus.cc:686] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:38.626685 9025 raft_consensus.cc:2726] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:27:38.626814 6644 leader_election.cc:302] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:38.626845 8274 consensus_queue.cc:237] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.627018 9027 raft_consensus.cc:2726] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:27:38.629891 6433 catalog_manager.cc:5484] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 2 to 3, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 3 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:38.642936 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b2b3e30b94d47dd822539cb30f36f81"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:38.643206 6839 raft_consensus.cc:482] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.643321 6839 raft_consensus.cc:3037] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.646001 6839 raft_consensus.cc:504] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.646889 6839 leader_election.cc:288] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 4 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:38.647250 6710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2b3e30b94d47dd822539cb30f36f81" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.647486 6710 raft_consensus.cc:3032] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:38.647580 6710 raft_consensus.cc:729] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:38.647815 6710 consensus_queue.cc:260] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.648065 6710 raft_consensus.cc:3037] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.650460 6710 raft_consensus.cc:2396] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.650856 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2b3e30b94d47dd822539cb30f36f81" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.651052 6968 raft_consensus.cc:3037] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.653419 6968 raft_consensus.cc:2396] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.654208 6776 leader_election.cc:302] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720; no voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:38.654449 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "df917e7196ed4939b75dc52930c87640"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:38.654716 8281 raft_consensus.cc:2726] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:27:38.654742 6839 raft_consensus.cc:482] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.654979 6839 raft_consensus.cc:3037] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.658267 6839 raft_consensus.cc:504] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.659539 6710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df917e7196ed4939b75dc52930c87640" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.659768 6710 raft_consensus.cc:2379] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 in current term 3: Already voted for candidate ab5c055dc24f4944a58d7d2593ddc8ef in this term.
I20240502 21:27:38.659976 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df917e7196ed4939b75dc52930c87640" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.660660 6839 leader_election.cc:288] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 3 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:38.663391 6579 raft_consensus.cc:2454] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 3.
I20240502 21:27:38.663879 6774 leader_election.cc:302] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters: ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:38.664371 8281 raft_consensus.cc:2781] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:38.667987 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3619e8a307de4583a4e470ad583fd7bc"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:38.668552 6839 raft_consensus.cc:482] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.669037 6839 raft_consensus.cc:3037] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.672731 8281 raft_consensus.cc:686] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 3 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:38.673242 8281 consensus_queue.cc:237] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.673332 6839 raft_consensus.cc:504] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:38.673933 6839 leader_election.cc:288] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 4 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:38.674254 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3619e8a307de4583a4e470ad583fd7bc" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.674444 6968 raft_consensus.cc:3032] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:38.674539 6968 raft_consensus.cc:729] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.674587 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3619e8a307de4583a4e470ad583fd7bc" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.674772 6579 raft_consensus.cc:3037] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.674808 6968 consensus_queue.cc:260] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:38.675096 6968 raft_consensus.cc:3037] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.677274 6436 catalog_manager.cc:5484] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 2 to 3, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 3 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:38.678005 6968 raft_consensus.cc:2396] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.678588 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:38.678731 6839 raft_consensus.cc:482] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.678828 6839 raft_consensus.cc:3037] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.680971 6839 raft_consensus.cc:504] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.681721 6839 leader_election.cc:288] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 4 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:38.682138 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.682286 6577 raft_consensus.cc:3037] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.684177 6577 raft_consensus.cc:2396] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.684943 6840 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c10bca264dda4180bd19beb5767bf047"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:38.685005 6710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.685110 6840 raft_consensus.cc:482] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.685169 6710 raft_consensus.cc:3032] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:38.685212 6840 raft_consensus.cc:3037] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.685267 6710 raft_consensus.cc:729] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:38.685488 6710 consensus_queue.cc:260] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.685792 6710 raft_consensus.cc:3037] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.687028 6840 raft_consensus.cc:504] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.687556 6840 leader_election.cc:288] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 4 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:38.688113 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10bca264dda4180bd19beb5767bf047" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.688293 6968 raft_consensus.cc:3037] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.690747 6968 raft_consensus.cc:2396] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.691356 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10bca264dda4180bd19beb5767bf047" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.691586 6711 raft_consensus.cc:3032] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:38.691670 6711 raft_consensus.cc:729] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:38.691926 6711 consensus_queue.cc:260] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.692054 6710 raft_consensus.cc:2396] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.692183 6711 raft_consensus.cc:3037] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.693476 6579 raft_consensus.cc:2396] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.694634 6711 raft_consensus.cc:2396] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.694614 6840 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "66198d766e6e4ceb820bc92bcca67d85"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:38.694790 6840 raft_consensus.cc:482] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.694875 6840 raft_consensus.cc:3037] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.695605 6776 leader_election.cc:302] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720; no voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:38.696097 6776 leader_election.cc:302] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720; no voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:38.696238 6774 leader_election.cc:302] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720; no voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:38.696674 9074 raft_consensus.cc:2726] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:27:38.696799 9074 raft_consensus.cc:2726] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:27:38.696903 9074 raft_consensus.cc:2726] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:27:38.697337 6840 raft_consensus.cc:504] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:38.698395 6840 leader_election.cc:288] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 4 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:38.698623 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66198d766e6e4ceb820bc92bcca67d85" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.698824 6968 raft_consensus.cc:3032] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:38.698909 6968 raft_consensus.cc:729] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.699635 6968 consensus_queue.cc:260] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:38.699760 6840 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "980f9cc0cdc6452d8009e9f75a0d4275"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:38.699904 6840 raft_consensus.cc:482] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.699923 6968 raft_consensus.cc:3037] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.700022 6840 raft_consensus.cc:3037] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.702472 6968 raft_consensus.cc:2396] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.702373 6840 raft_consensus.cc:504] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.702920 6840 leader_election.cc:288] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 4 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:38.703320 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "980f9cc0cdc6452d8009e9f75a0d4275" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.703483 6970 raft_consensus.cc:3037] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.705682 6970 raft_consensus.cc:2396] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.707276 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "980f9cc0cdc6452d8009e9f75a0d4275" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.707518 6711 raft_consensus.cc:3032] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:38.707595 6711 raft_consensus.cc:729] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:38.707888 6711 consensus_queue.cc:260] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.708026 6840 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f0b941baaab542a78748d4d0894cfbf4"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:38.708163 6711 raft_consensus.cc:3037] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.708344 6840 raft_consensus.cc:482] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.708525 6840 raft_consensus.cc:3037] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.711174 6711 raft_consensus.cc:2396] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.711576 6776 leader_election.cc:302] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720; no voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:38.712107 8281 raft_consensus.cc:2726] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:27:38.713680 6840 raft_consensus.cc:504] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.714597 6840 leader_election.cc:288] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 4 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:38.714823 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b941baaab542a78748d4d0894cfbf4" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.715158 6711 raft_consensus.cc:3032] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:38.715241 6711 raft_consensus.cc:729] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:38.715517 6840 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3b6b526ec34c41aea64a2a19f0208f5d"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:38.715531 6711 consensus_queue.cc:260] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.715818 6711 raft_consensus.cc:3037] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.715818 6840 raft_consensus.cc:482] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.715991 6840 raft_consensus.cc:3037] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.718757 6711 raft_consensus.cc:2396] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:27:38.719415 6840 raft_consensus.cc:504] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.720396 6840 leader_election.cc:288] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 3 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:38.720659 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6b526ec34c41aea64a2a19f0208f5d" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.720861 6970 raft_consensus.cc:3032] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:38.720945 6970 raft_consensus.cc:729] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.721195 6970 consensus_queue.cc:260] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.721464 6970 raft_consensus.cc:3037] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.723846 6970 raft_consensus.cc:2396] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.724501 6840 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "da14dcc58d684704b3649e85b26fb43c"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:38.724730 6840 raft_consensus.cc:482] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.724889 6840 raft_consensus.cc:3037] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.727283 6840 raft_consensus.cc:504] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:38.728283 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da14dcc58d684704b3649e85b26fb43c" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.728566 6711 raft_consensus.cc:3032] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:38.729339 6839 raft_consensus.cc:1226] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Rejecting Update request from peer 80a838c00b544bf68ec4cc9c513212b2 for earlier term 2. Current term is 3. Ops: []
I20240502 21:27:38.729410 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6b526ec34c41aea64a2a19f0208f5d" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.729632 6580 raft_consensus.cc:3037] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.729732 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b941baaab542a78748d4d0894cfbf4" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.729945 6577 raft_consensus.cc:3037] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.732297 6580 raft_consensus.cc:2454] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 3.
I20240502 21:27:38.732647 6577 raft_consensus.cc:2454] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 4.
I20240502 21:27:38.732782 6711 raft_consensus.cc:729] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:38.732952 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66198d766e6e4ceb820bc92bcca67d85" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.733136 6579 raft_consensus.cc:3037] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.733381 6711 consensus_queue.cc:260] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:38.733685 6711 raft_consensus.cc:3037] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.734131 6840 leader_election.cc:288] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 4 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:38.734908 6774 leader_election.cc:302] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters: 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:38.735291 6774 leader_election.cc:302] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters: ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:38.735471 6579 raft_consensus.cc:2396] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.735615 8281 raft_consensus.cc:2781] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:38.735759 8281 raft_consensus.cc:686] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 3 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:38.735949 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da14dcc58d684704b3649e85b26fb43c" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.736128 6577 raft_consensus.cc:3037] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.736024 8281 consensus_queue.cc:237] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.736548 9074 raft_consensus.cc:2781] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:27:38.737031 6711 raft_consensus.cc:2396] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.739482 6577 raft_consensus.cc:2396] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.740368 6840 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5e38d544ee14490c9fb5d8069440fd6a"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:38.740854 6840 raft_consensus.cc:482] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.741478 6774 leader_election.cc:302] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720; no voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:38.741861 9074 raft_consensus.cc:686] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 4 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:38.741880 6774 leader_election.cc:302] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720; no voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:38.741968 8281 raft_consensus.cc:2726] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:27:38.742115 8281 raft_consensus.cc:2726] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:27:38.742163 9074 consensus_queue.cc:237] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.743149 6840 raft_consensus.cc:3037] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.745369 6840 raft_consensus.cc:504] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.745683 6432 catalog_manager.cc:5484] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 2 to 3, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 3 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.746279 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e38d544ee14490c9fb5d8069440fd6a" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.746476 6970 raft_consensus.cc:3032] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:38.746289 6432 catalog_manager.cc:5484] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 3 to 4, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 4 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.746529 6840 leader_election.cc:288] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 4 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:38.746555 6970 raft_consensus.cc:729] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.746809 6970 consensus_queue.cc:260] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.747078 6970 raft_consensus.cc:3037] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.747504 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e38d544ee14490c9fb5d8069440fd6a" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.747720 6577 raft_consensus.cc:3037] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.749836 6970 raft_consensus.cc:2396] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.750514 6577 raft_consensus.cc:2396] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.750844 6774 leader_election.cc:302] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720; no voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:38.751053 6840 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a09d2491d9dc4cbfb20f7adc1087b375"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:38.751242 6840 raft_consensus.cc:482] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.751358 6840 raft_consensus.cc:3037] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.758399 9074 raft_consensus.cc:2726] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:27:38.764569 6840 raft_consensus.cc:504] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.765568 6840 leader_election.cc:288] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 4 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:38.765916 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09d2491d9dc4cbfb20f7adc1087b375" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.766114 6970 raft_consensus.cc:3037] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.766266 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09d2491d9dc4cbfb20f7adc1087b375" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.766521 6711 raft_consensus.cc:3032] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:38.766599 6711 raft_consensus.cc:729] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:38.766876 6711 consensus_queue.cc:260] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.767169 6711 raft_consensus.cc:3037] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.769168 6840 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5b2e443bad344ce89ce69871406c5a29"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:38.769320 6840 raft_consensus.cc:482] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.769430 6840 raft_consensus.cc:3037] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.769811 6970 raft_consensus.cc:2396] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.770680 6711 raft_consensus.cc:2396] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.771193 6776 leader_election.cc:302] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720; no voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:38.771451 9074 raft_consensus.cc:2726] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:27:38.782606 6840 raft_consensus.cc:504] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.784045 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2e443bad344ce89ce69871406c5a29" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.784101 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2e443bad344ce89ce69871406c5a29" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.784265 6970 raft_consensus.cc:3032] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:38.784255 6577 raft_consensus.cc:3037] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.784354 6970 raft_consensus.cc:729] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.784669 6970 consensus_queue.cc:260] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.784930 6970 raft_consensus.cc:3037] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.786684 6577 raft_consensus.cc:2396] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.787375 6970 raft_consensus.cc:2396] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.787967 6775 leader_election.cc:302] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720; no voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:38.788226 9074 raft_consensus.cc:2726] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:27:38.788614 6840 leader_election.cc:288] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 4 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:38.789346 6558 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:38.793812 6840 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57e37f7e15b74819b692792eaed94f24"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:38.794078 6840 raft_consensus.cc:482] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.794245 6840 raft_consensus.cc:3037] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.805117 8750 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:38.814069 6840 raft_consensus.cc:504] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.815356 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e37f7e15b74819b692792eaed94f24" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.815533 6970 raft_consensus.cc:3037] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.818174 6970 raft_consensus.cc:2454] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 3.
I20240502 21:27:38.818636 6775 leader_election.cc:302] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:38.819113 9074 raft_consensus.cc:2781] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:38.819286 9074 raft_consensus.cc:686] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [term 3 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:38.819197 6840 leader_election.cc:288] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 3 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:38.819428 8485 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:38.819547 9074 consensus_queue.cc:237] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.820222 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e37f7e15b74819b692792eaed94f24" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.820518 6711 raft_consensus.cc:3032] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:38.820612 6711 raft_consensus.cc:729] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:38.820899 6711 consensus_queue.cc:260] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.821202 6711 raft_consensus.cc:3037] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.824205 6711 raft_consensus.cc:2396] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.827994 6840 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fbec2b1b80644a93b9239600157ad957"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:38.828295 6840 raft_consensus.cc:482] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.828507 6840 raft_consensus.cc:3037] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.831454 6690 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:27:38.851421 6840 raft_consensus.cc:504] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.852596 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec2b1b80644a93b9239600157ad957" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.852849 6577 raft_consensus.cc:3037] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.855424 6577 raft_consensus.cc:2454] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 3.
I20240502 21:27:38.855912 6774 leader_election.cc:302] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:38.856213 9079 raft_consensus.cc:2781] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:38.856356 9079 raft_consensus.cc:686] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 3 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:38.856637 9079 consensus_queue.cc:237] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.866259 6840 leader_election.cc:288] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 3 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:38.866604 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec2b1b80644a93b9239600157ad957" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.866875 6970 raft_consensus.cc:3032] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:38.866976 6970 raft_consensus.cc:729] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:38.867298 6970 consensus_queue.cc:260] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.867626 6970 raft_consensus.cc:3037] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.874445 6970 raft_consensus.cc:2396] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.874927 6840 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c6d11cba6ca04c17a9f47506824d9aa2"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:38.875201 6840 raft_consensus.cc:482] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.875351 6840 raft_consensus.cc:3037] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.879686 6432 catalog_manager.cc:5484] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 2 to 3, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 3 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:38.882661 6577 raft_consensus.cc:1226] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Rejecting Update request from peer 80a838c00b544bf68ec4cc9c513212b2 for earlier term 2. Current term is 3. Ops: []
I20240502 21:27:38.885939 6432 catalog_manager.cc:5484] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 2 to 3, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 3 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:38.887393 8618 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:38.900877 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:38.884722 6840 raft_consensus.cc:504] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.902248 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d11cba6ca04c17a9f47506824d9aa2" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.902467 6577 raft_consensus.cc:3037] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.905172 6577 raft_consensus.cc:2396] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.906311 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d11cba6ca04c17a9f47506824d9aa2" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.906522 6711 raft_consensus.cc:3032] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:38.906587 6711 raft_consensus.cc:729] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:38.906867 6711 consensus_queue.cc:260] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:38.907135 6711 raft_consensus.cc:3037] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.909978 6711 raft_consensus.cc:2396] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.910024 6820 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:38.910588 6776 leader_election.cc:302] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720; no voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:38.910917 6840 leader_election.cc:288] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 4 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:38.913260 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "444ccd95dc0f4bdd806ca9151d56d5cd"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:38.913414 6839 raft_consensus.cc:482] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.913517 6839 raft_consensus.cc:3037] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.916184 6839 raft_consensus.cc:504] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.917016 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444ccd95dc0f4bdd806ca9151d56d5cd" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.917186 9074 raft_consensus.cc:2726] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:27:38.917237 6711 raft_consensus.cc:3032] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:38.917234 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444ccd95dc0f4bdd806ca9151d56d5cd" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.917316 6711 raft_consensus.cc:729] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:38.917469 6970 raft_consensus.cc:3037] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.917614 6711 consensus_queue.cc:260] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.917737 8948 heartbeater.cc:503] Master 127.24.226.190:40767 was elected leader, sending a full tablet report...
I20240502 21:27:38.917878 6711 raft_consensus.cc:3037] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.920010 6970 raft_consensus.cc:2454] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 3.
I20240502 21:27:38.920939 6577 raft_consensus.cc:1261] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Refusing update from remote peer 80a838c00b544bf68ec4cc9c513212b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:27:38.922035 6711 raft_consensus.cc:2396] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.922654 6840 raft_consensus.cc:1261] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Refusing update from remote peer 80a838c00b544bf68ec4cc9c513212b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:27:38.922901 6775 leader_election.cc:302] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters: ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:38.923270 9002 consensus_queue.cc:1035] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:38.924291 9079 raft_consensus.cc:2781] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:38.924455 9079 raft_consensus.cc:686] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [term 3 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:38.924721 9079 consensus_queue.cc:237] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:38.930714 9002 consensus_queue.cc:1035] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:38.936666 6839 leader_election.cc:288] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 3 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:38.938153 6841 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3093485f30b41939a1380ab7aa9e38b"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:38.938375 6841 raft_consensus.cc:482] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.938469 6949 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:38.938534 6841 raft_consensus.cc:3037] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.941562 6841 raft_consensus.cc:504] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.942155 6841 leader_election.cc:288] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 4 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:38.942489 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3093485f30b41939a1380ab7aa9e38b" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.942663 6970 raft_consensus.cc:3037] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.942821 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3093485f30b41939a1380ab7aa9e38b" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.943081 6711 raft_consensus.cc:3032] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:38.943166 6711 raft_consensus.cc:729] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:38.943475 6711 consensus_queue.cc:260] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.943749 6711 raft_consensus.cc:3037] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.942998 6839 raft_consensus.cc:1226] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Rejecting Update request from peer ab5c055dc24f4944a58d7d2593ddc8ef for earlier term 3. Current term is 4. Ops: []
I20240502 21:27:38.945344 6970 raft_consensus.cc:2396] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.945276 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1c98cbfbd0a44335b75b228c217f9132"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:38.945513 6839 raft_consensus.cc:482] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.945686 6839 raft_consensus.cc:3037] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.946545 6711 raft_consensus.cc:2396] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.946974 6776 leader_election.cc:302] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720; no voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:38.947296 9079 raft_consensus.cc:2726] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:27:38.951031 6432 catalog_manager.cc:5484] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 2 to 3, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 3 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:38.965880 6577 raft_consensus.cc:1261] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Refusing update from remote peer ab5c055dc24f4944a58d7d2593ddc8ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 21:27:38.966459 8274 consensus_queue.cc:1035] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:38.970494 6839 raft_consensus.cc:504] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:38.971335 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c98cbfbd0a44335b75b228c217f9132" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.971531 6711 raft_consensus.cc:3032] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:38.971505 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c98cbfbd0a44335b75b228c217f9132" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.971611 6711 raft_consensus.cc:729] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:38.971837 6711 consensus_queue.cc:260] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:38.972013 6839 leader_election.cc:288] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 3 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:38.972083 6711 raft_consensus.cc:3037] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.972882 6841 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ea3b52c673ae4b9f931059c4b0a456fa"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:38.973054 6841 raft_consensus.cc:482] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.973145 6841 raft_consensus.cc:3037] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.974560 6711 raft_consensus.cc:2396] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.974907 6776 leader_election.cc:302] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720; no voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:38.975281 9079 raft_consensus.cc:2726] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 21:27:38.975581 6841 raft_consensus.cc:504] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.977272 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3b52c673ae4b9f931059c4b0a456fa" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:38.977519 6970 raft_consensus.cc:3037] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.980257 6970 raft_consensus.cc:2396] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.981292 6841 leader_election.cc:288] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 4 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:38.981534 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3b52c673ae4b9f931059c4b0a456fa" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.981729 6711 raft_consensus.cc:3032] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:38.981825 6711 raft_consensus.cc:729] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:38.982088 6711 consensus_queue.cc:260] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.982364 6711 raft_consensus.cc:3037] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.982343 6841 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "169899da82594d9cb673f8db74764ca2"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:38.982594 6841 raft_consensus.cc:482] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.982749 6841 raft_consensus.cc:3037] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.985100 6711 raft_consensus.cc:2396] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.985487 6776 leader_election.cc:302] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720; no voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:38.985847 9079 raft_consensus.cc:2726] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:27:38.985231 6841 raft_consensus.cc:504] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.986733 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169899da82594d9cb673f8db74764ca2" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:38.986934 6579 raft_consensus.cc:3037] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.989777 6579 raft_consensus.cc:2396] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.991029 6841 leader_election.cc:288] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 4 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:38.991101 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169899da82594d9cb673f8db74764ca2" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:38.991283 6711 raft_consensus.cc:3032] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:38.991357 6711 raft_consensus.cc:729] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:38.991590 6711 consensus_queue.cc:260] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:38.991871 6711 raft_consensus.cc:3037] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:38.991914 6841 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8fac59b25c164e658fe71cfdf7ca78cb"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:38.992074 6841 raft_consensus.cc:482] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:38.992158 6841 raft_consensus.cc:3037] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:38.994621 6711 raft_consensus.cc:2396] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:38.995077 6776 leader_election.cc:302] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720; no voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:38.995414 9079 raft_consensus.cc:2726] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:27:38.998870 6970 raft_consensus.cc:1261] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Refusing update from remote peer ab5c055dc24f4944a58d7d2593ddc8ef: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 21:27:38.998984 6970 pending_rounds.cc:76] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2: Aborting all ops after (but not including) 3
I20240502 21:27:38.999094 6970 pending_rounds.cc:97] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 21:27:38.999230 6970 raft_consensus.cc:2866] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:27:39.000900 8274 consensus_queue.cc:1035] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:39.001711 6841 raft_consensus.cc:504] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:39.002684 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fac59b25c164e658fe71cfdf7ca78cb" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:39.002898 6711 raft_consensus.cc:3032] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:39.002979 6711 raft_consensus.cc:729] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:39.003319 6711 consensus_queue.cc:260] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:39.003602 6711 raft_consensus.cc:3037] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:39.006628 6711 raft_consensus.cc:2396] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:39.008963 6841 leader_election.cc:288] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 3 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:39.009369 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fac59b25c164e658fe71cfdf7ca78cb" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:39.009550 6580 raft_consensus.cc:3037] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:39.012156 6580 raft_consensus.cc:2454] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 3.
I20240502 21:27:39.013031 6774 leader_election.cc:302] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters: ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:39.013418 9079 raft_consensus.cc:2781] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:39.013530 9079 raft_consensus.cc:686] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [term 3 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:39.013767 9079 consensus_queue.cc:237] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:39.017606 6435 catalog_manager.cc:5484] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 2 to 3, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 3 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.024698 6841 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "093f8fd73ec34d8b9954c1716e910a78"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:39.024919 6841 raft_consensus.cc:482] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.025338 6841 raft_consensus.cc:3037] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:39.028223 6841 raft_consensus.cc:504] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.028934 6579 raft_consensus.cc:1261] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Refusing update from remote peer ab5c055dc24f4944a58d7d2593ddc8ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 21:27:39.029330 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093f8fd73ec34d8b9954c1716e910a78" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:39.029419 9025 consensus_queue.cc:1035] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:39.029527 6580 raft_consensus.cc:3037] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:39.032157 6580 raft_consensus.cc:2454] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 3.
I20240502 21:27:39.032595 6774 leader_election.cc:302] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:39.032876 9079 raft_consensus.cc:2781] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:39.033147 9079 raft_consensus.cc:686] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 3 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:39.033439 9079 consensus_queue.cc:237] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.035322 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093f8fd73ec34d8b9954c1716e910a78" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:39.035529 6970 raft_consensus.cc:3032] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:27:39.035634 6970 raft_consensus.cc:729] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.035872 6970 consensus_queue.cc:260] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.036262 6970 raft_consensus.cc:3037] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:39.036209 6435 catalog_manager.cc:5484] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 2 to 3, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 3 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:39.038393 6970 raft_consensus.cc:2454] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 3.
I20240502 21:27:39.034896 6841 leader_election.cc:288] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 3 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:39.066092 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "df917e7196ed4939b75dc52930c87640"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:39.066282 6577 raft_consensus.cc:482] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.066403 6577 raft_consensus.cc:3037] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:39.069502 6577 raft_consensus.cc:504] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.070178 6577 leader_election.cc:288] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 4 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:39.070537 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df917e7196ed4939b75dc52930c87640" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:39.070631 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df917e7196ed4939b75dc52930c87640" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.070761 6711 raft_consensus.cc:3037] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:39.070822 6841 raft_consensus.cc:3032] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:39.070887 6841 raft_consensus.cc:729] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:39.071480 6841 consensus_queue.cc:260] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.071916 6841 raft_consensus.cc:3037] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:39.073304 6711 raft_consensus.cc:2454] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 4.
I20240502 21:27:39.074651 6515 leader_election.cc:302] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:39.074973 6970 raft_consensus.cc:1261] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Refusing update from remote peer ab5c055dc24f4944a58d7d2593ddc8ef: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 21:27:39.075075 6970 pending_rounds.cc:76] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2: Aborting all ops after (but not including) 3
I20240502 21:27:39.075142 6970 pending_rounds.cc:97] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 21:27:39.075198 6970 raft_consensus.cc:2866] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:27:39.075543 9113 raft_consensus.cc:2781] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:27:39.075654 9113 raft_consensus.cc:686] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 4 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.075883 9113 consensus_queue.cc:237] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.077224 6580 raft_consensus.cc:1261] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Refusing update from remote peer ab5c055dc24f4944a58d7d2593ddc8ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 21:27:39.077345 6841 raft_consensus.cc:2396] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:27:39.078043 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:39.078022 9025 consensus_queue.cc:1035] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:39.078235 6577 raft_consensus.cc:482] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.078331 6577 raft_consensus.cc:3037] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.078608 9025 consensus_queue.cc:1035] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:39.085103 6577 raft_consensus.cc:504] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.086164 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:39.086342 6711 raft_consensus.cc:3037] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.089609 6711 raft_consensus.cc:2396] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:27:39.091614 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.091845 6841 raft_consensus.cc:3037] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.094337 6841 raft_consensus.cc:2454] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 5.
I20240502 21:27:39.094722 6513 leader_election.cc:302] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters: ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:39.094954 8965 raft_consensus.cc:2781] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:27:39.095178 6577 leader_election.cc:288] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 5 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:39.096170 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3619e8a307de4583a4e470ad583fd7bc"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:39.096330 6577 raft_consensus.cc:482] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.096941 6577 raft_consensus.cc:3037] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.099684 6577 raft_consensus.cc:504] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:39.100615 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3619e8a307de4583a4e470ad583fd7bc" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:39.100847 6970 raft_consensus.cc:3037] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.103909 6970 raft_consensus.cc:2396] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:27:39.103847 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3619e8a307de4583a4e470ad583fd7bc" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.104018 6841 raft_consensus.cc:3037] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.106462 6841 raft_consensus.cc:2454] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 5.
I20240502 21:27:39.106843 6513 leader_election.cc:302] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters: 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:39.108317 9124 raft_consensus.cc:2781] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:27:39.108585 8965 raft_consensus.cc:686] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.108999 8965 consensus_queue.cc:237] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.110096 9124 raft_consensus.cc:686] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.110483 9124 consensus_queue.cc:237] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:39.117854 6577 leader_election.cc:288] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 5 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:39.119346 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aae0bc634aa7424e935c8bc81a85099b"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:39.119527 6577 raft_consensus.cc:482] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.119627 6577 raft_consensus.cc:3037] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.122643 6577 raft_consensus.cc:504] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.123528 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae0bc634aa7424e935c8bc81a85099b" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:39.123739 6711 raft_consensus.cc:3037] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.123986 6435 catalog_manager.cc:5484] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 3 to 4, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 4 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.126196 6711 raft_consensus.cc:2454] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 5.
I20240502 21:27:39.126603 6515 leader_election.cc:302] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:39.126822 6577 leader_election.cc:288] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 5 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:39.126930 8965 raft_consensus.cc:2781] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:27:39.127534 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae0bc634aa7424e935c8bc81a85099b" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:39.127689 8965 raft_consensus.cc:686] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.127738 6970 raft_consensus.cc:3037] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.127722 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3b6b526ec34c41aea64a2a19f0208f5d"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:39.127893 6577 raft_consensus.cc:482] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.128026 6577 raft_consensus.cc:3037] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:39.127961 8965 consensus_queue.cc:237] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.130674 6970 raft_consensus.cc:2396] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:27:39.130628 6577 raft_consensus.cc:504] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.131179 6577 leader_election.cc:288] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 4 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:39.131582 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6b526ec34c41aea64a2a19f0208f5d" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:39.131754 6968 raft_consensus.cc:3037] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:39.133514 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6b526ec34c41aea64a2a19f0208f5d" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.133689 6841 raft_consensus.cc:3032] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:39.133751 6841 raft_consensus.cc:729] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:39.133975 6841 consensus_queue.cc:260] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.134274 6968 raft_consensus.cc:2396] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:39.134290 6841 raft_consensus.cc:3037] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:39.135519 6435 catalog_manager.cc:5484] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 3 to 5, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 5 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:39.135989 6435 catalog_manager.cc:5484] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 3 to 5, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 5 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.137881 6841 raft_consensus.cc:2396] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:39.140113 6580 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "da14dcc58d684704b3649e85b26fb43c"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:39.140295 6580 raft_consensus.cc:482] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.140395 6580 raft_consensus.cc:3037] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.140463 6514 leader_election.cc:302] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae; no voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720
I20240502 21:27:39.141983 6577 raft_consensus.cc:1261] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Refusing update from remote peer 9793130a8e4f45388357b8d2e9f47720: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240502 21:27:39.142448 8965 raft_consensus.cc:2726] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:27:39.142685 6580 raft_consensus.cc:504] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:39.143162 6580 leader_election.cc:288] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 5 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:39.143484 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da14dcc58d684704b3649e85b26fb43c" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:39.143608 9079 consensus_queue.cc:1035] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:39.143669 6711 raft_consensus.cc:3037] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.144846 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da14dcc58d684704b3649e85b26fb43c" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.145035 6841 raft_consensus.cc:3037] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.146087 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "66198d766e6e4ceb820bc92bcca67d85"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:39.146270 6577 raft_consensus.cc:482] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.146545 6711 raft_consensus.cc:2396] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:27:39.146593 6577 raft_consensus.cc:3037] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.147742 6841 raft_consensus.cc:2454] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 5.
I20240502 21:27:39.148918 6435 catalog_manager.cc:5484] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 3 to 5, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 5 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.150108 6513 leader_election.cc:302] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters: ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:39.150750 9113 raft_consensus.cc:2781] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:27:39.150893 9113 raft_consensus.cc:686] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.151109 9113 consensus_queue.cc:237] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:39.149662 6577 raft_consensus.cc:504] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:39.153090 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66198d766e6e4ceb820bc92bcca67d85" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:39.153319 6968 raft_consensus.cc:3037] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.155673 6711 raft_consensus.cc:1261] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Refusing update from remote peer 9793130a8e4f45388357b8d2e9f47720: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20240502 21:27:39.155781 6711 pending_rounds.cc:76] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef: Aborting all ops after (but not including) 4
I20240502 21:27:39.155879 6968 raft_consensus.cc:2396] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:27:39.155850 6711 pending_rounds.cc:97] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20240502 21:27:39.155967 6711 raft_consensus.cc:2866] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:27:39.156868 8281 consensus_queue.cc:1035] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:39.160110 6577 leader_election.cc:288] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 5 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:39.160547 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66198d766e6e4ceb820bc92bcca67d85" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.160852 6841 raft_consensus.cc:3037] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.162801 6432 catalog_manager.cc:5484] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 3 to 5, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 5 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:39.163188 6841 raft_consensus.cc:2454] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 5.
I20240502 21:27:39.163537 6513 leader_election.cc:302] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters: 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:39.163959 9113 raft_consensus.cc:2781] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:27:39.164080 9113 raft_consensus.cc:686] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.164319 9113 consensus_queue.cc:237] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:39.167897 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f0b941baaab542a78748d4d0894cfbf4"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:39.168423 6577 raft_consensus.cc:482] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.168654 6577 raft_consensus.cc:3037] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.172232 6577 raft_consensus.cc:504] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.173485 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b941baaab542a78748d4d0894cfbf4" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.173740 6841 raft_consensus.cc:3032] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 21:27:39.173822 6841 raft_consensus.cc:729] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:39.174147 6841 consensus_queue.cc:260] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.174508 6841 raft_consensus.cc:3037] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.178350 6841 raft_consensus.cc:2454] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 5.
I20240502 21:27:39.178927 6513 leader_election.cc:302] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:39.179272 8965 raft_consensus.cc:2781] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:27:39.179420 8965 raft_consensus.cc:686] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.179490 6577 leader_election.cc:288] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 5 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:39.179659 8965 consensus_queue.cc:237] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.179860 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b941baaab542a78748d4d0894cfbf4" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:39.180065 6711 raft_consensus.cc:3037] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.182421 6711 raft_consensus.cc:2454] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 5.
I20240502 21:27:39.182689 6435 catalog_manager.cc:5484] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 4 to 5, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 5 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.183236 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5e38d544ee14490c9fb5d8069440fd6a"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:39.183382 6577 raft_consensus.cc:482] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.183212 6435 catalog_manager.cc:5484] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 3 to 5, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 5 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:39.183488 6577 raft_consensus.cc:3037] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.185343 6577 raft_consensus.cc:504] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.185892 6577 leader_election.cc:288] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 5 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:39.186475 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e38d544ee14490c9fb5d8069440fd6a" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:39.186501 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e38d544ee14490c9fb5d8069440fd6a" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.186677 6968 raft_consensus.cc:3037] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.186738 6841 raft_consensus.cc:3037] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.186802 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5b2e443bad344ce89ce69871406c5a29"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:39.186959 6577 raft_consensus.cc:482] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.187047 6577 raft_consensus.cc:3037] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.189567 6968 raft_consensus.cc:2396] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:27:39.189594 6577 raft_consensus.cc:504] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.190129 6577 leader_election.cc:288] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 5 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:39.190157 6841 raft_consensus.cc:2454] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 5.
I20240502 21:27:39.190601 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2e443bad344ce89ce69871406c5a29" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.190807 6840 raft_consensus.cc:3037] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.190958 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fb023d72aa8d4248a64d4723e18a3cf6"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:39.191099 6577 raft_consensus.cc:482] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.191201 6577 raft_consensus.cc:3037] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.193392 6840 raft_consensus.cc:2454] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 5.
I20240502 21:27:39.193519 6577 raft_consensus.cc:504] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.194160 6513 leader_election.cc:302] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters: 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:39.194160 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2e443bad344ce89ce69871406c5a29" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:39.194257 6577 leader_election.cc:288] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 5 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:39.194363 6970 raft_consensus.cc:3037] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.194463 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb023d72aa8d4248a64d4723e18a3cf6" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:39.194562 6513 leader_election.cc:302] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:39.194633 6711 raft_consensus.cc:3037] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.194811 9126 raft_consensus.cc:2781] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:27:39.194937 9126 raft_consensus.cc:686] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.195243 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb023d72aa8d4248a64d4723e18a3cf6" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:39.195205 9126 consensus_queue.cc:237] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.195399 6969 raft_consensus.cc:3037] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.196200 8965 raft_consensus.cc:2781] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:27:39.196342 8965 raft_consensus.cc:686] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.196576 8965 consensus_queue.cc:237] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.197082 6970 raft_consensus.cc:2396] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:27:39.197053 6711 raft_consensus.cc:2454] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 5.
I20240502 21:27:39.197955 6515 leader_election.cc:302] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:39.198112 6969 raft_consensus.cc:2396] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:27:39.198251 9124 raft_consensus.cc:2781] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:27:39.198405 9124 raft_consensus.cc:686] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.198657 9124 consensus_queue.cc:237] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.201372 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c6d11cba6ca04c17a9f47506824d9aa2"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:39.201620 6577 raft_consensus.cc:482] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.201751 6577 raft_consensus.cc:3037] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.203403 6435 catalog_manager.cc:5484] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 3 to 5, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 5 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.204133 6577 raft_consensus.cc:504] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.203923 6435 catalog_manager.cc:5484] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 3 to 5, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 5 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.204727 6577 leader_election.cc:288] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 5 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:39.204535 6435 catalog_manager.cc:5484] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 3 to 5, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 5 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.205139 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d11cba6ca04c17a9f47506824d9aa2" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:39.205338 6711 raft_consensus.cc:3037] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.206033 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d11cba6ca04c17a9f47506824d9aa2" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.206210 6840 raft_consensus.cc:3037] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.207782 6711 raft_consensus.cc:2396] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:27:39.208158 6840 raft_consensus.cc:2454] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 5.
I20240502 21:27:39.208616 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "331bcde3e6bf43c081af836c8cdebdc7"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:39.208674 6513 leader_election.cc:302] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters: ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:39.208940 9126 raft_consensus.cc:2781] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:27:39.209208 9126 raft_consensus.cc:686] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.209445 9126 consensus_queue.cc:237] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.211035 6577 raft_consensus.cc:482] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.211153 6577 raft_consensus.cc:3037] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:39.213073 6435 catalog_manager.cc:5484] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 3 to 5, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 5 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.213259 6577 raft_consensus.cc:504] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.213843 6577 leader_election.cc:288] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 4 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:39.214573 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331bcde3e6bf43c081af836c8cdebdc7" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:39.214756 6711 raft_consensus.cc:3032] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:39.214756 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331bcde3e6bf43c081af836c8cdebdc7" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:39.214865 6711 raft_consensus.cc:729] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:39.214943 6968 raft_consensus.cc:3037] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:39.215131 6711 consensus_queue.cc:260] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.215495 6711 raft_consensus.cc:3037] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:39.216387 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fbec2b1b80644a93b9239600157ad957"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:39.216573 6577 raft_consensus.cc:482] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.216655 6577 raft_consensus.cc:3037] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:39.217471 6968 raft_consensus.cc:2454] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 4.
I20240502 21:27:39.217921 6711 raft_consensus.cc:2454] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 4.
I20240502 21:27:39.218011 6514 leader_election.cc:302] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2; no voters:
I20240502 21:27:39.218606 9126 raft_consensus.cc:2781] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:27:39.218730 9126 raft_consensus.cc:686] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 4 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.218953 9126 consensus_queue.cc:237] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.219082 6577 raft_consensus.cc:504] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.219626 6577 leader_election.cc:288] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 4 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:39.220674 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec2b1b80644a93b9239600157ad957" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.220731 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1c98cbfbd0a44335b75b228c217f9132"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:39.220855 6840 raft_consensus.cc:3032] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:39.220885 6577 raft_consensus.cc:482] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.220927 6840 raft_consensus.cc:729] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:39.220981 6577 raft_consensus.cc:3037] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:27:39.221184 6840 consensus_queue.cc:260] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.221434 6840 raft_consensus.cc:3037] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:39.223178 6577 raft_consensus.cc:504] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:39.223778 6577 leader_election.cc:288] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 3 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:39.223940 6840 raft_consensus.cc:2396] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:39.224485 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c98cbfbd0a44335b75b228c217f9132" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:39.224974 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec2b1b80644a93b9239600157ad957" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:39.225167 6968 raft_consensus.cc:3037] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:39.225344 6435 catalog_manager.cc:5484] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 3 to 4, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 4 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.225663 6969 raft_consensus.cc:1261] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Refusing update from remote peer 9793130a8e4f45388357b8d2e9f47720: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 21:27:39.226207 9079 consensus_queue.cc:1035] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:39.226509 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "169899da82594d9cb673f8db74764ca2"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:39.226665 6577 raft_consensus.cc:482] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.226748 6577 raft_consensus.cc:3037] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.228392 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c98cbfbd0a44335b75b228c217f9132" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.228667 6840 raft_consensus.cc:2379] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae in current term 3: Already voted for candidate 9793130a8e4f45388357b8d2e9f47720 in this term.
I20240502 21:27:39.229856 6577 raft_consensus.cc:504] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.230237 6711 raft_consensus.cc:2454] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 3.
I20240502 21:27:39.230777 6710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169899da82594d9cb673f8db74764ca2" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:39.230823 6577 leader_election.cc:288] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 5 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:39.230890 6515 leader_election.cc:302] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 9793130a8e4f45388357b8d2e9f47720
I20240502 21:27:39.230934 6710 raft_consensus.cc:3037] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.231397 9124 raft_consensus.cc:2781] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:27:39.231516 9124 raft_consensus.cc:686] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 3 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.231531 6968 raft_consensus.cc:2396] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:39.231729 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8fac59b25c164e658fe71cfdf7ca78cb"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:39.231910 6577 raft_consensus.cc:482] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.231748 9124 consensus_queue.cc:237] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:39.231938 6514 leader_election.cc:302] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae; no voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720
I20240502 21:27:39.232036 6577 raft_consensus.cc:3037] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:39.232412 9113 raft_consensus.cc:2726] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:27:39.232703 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169899da82594d9cb673f8db74764ca2" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.232909 6840 raft_consensus.cc:3037] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.234262 6435 catalog_manager.cc:5484] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 2 to 3, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 3 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:39.235363 6577 raft_consensus.cc:504] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:39.236891 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fac59b25c164e658fe71cfdf7ca78cb" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.237053 6841 raft_consensus.cc:3032] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:39.237155 6841 raft_consensus.cc:729] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:39.237306 6577 leader_election.cc:288] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 4 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:39.237380 6841 consensus_queue.cc:260] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:39.237727 6841 raft_consensus.cc:3037] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:39.238756 6709 raft_consensus.cc:1261] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Refusing update from remote peer 9793130a8e4f45388357b8d2e9f47720: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 21:27:39.238833 6709 pending_rounds.cc:76] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef: Aborting all ops after (but not including) 3
I20240502 21:27:39.238896 6709 pending_rounds.cc:97] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 21:27:39.238945 6709 raft_consensus.cc:2866] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:27:39.239394 6840 raft_consensus.cc:2454] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 5.
I20240502 21:27:39.239845 8281 consensus_queue.cc:1035] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:39.240042 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fac59b25c164e658fe71cfdf7ca78cb" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:39.240463 6513 leader_election.cc:302] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:39.240705 9126 raft_consensus.cc:2781] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:27:39.240847 9126 raft_consensus.cc:686] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.241008 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "093f8fd73ec34d8b9954c1716e910a78"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:39.241075 9126 consensus_queue.cc:237] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.241169 6577 raft_consensus.cc:482] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.241246 6577 raft_consensus.cc:3037] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:39.241302 6710 raft_consensus.cc:2396] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:27:39.240227 6711 raft_consensus.cc:3037] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:39.242036 6841 raft_consensus.cc:2396] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:39.244212 6577 raft_consensus.cc:504] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.245335 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093f8fd73ec34d8b9954c1716e910a78" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:39.245540 6969 raft_consensus.cc:3037] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:39.246290 6711 raft_consensus.cc:2396] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:39.246625 6515 leader_election.cc:302] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae; no voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:39.246865 9124 raft_consensus.cc:2726] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 21:27:39.250450 6969 raft_consensus.cc:2454] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 4.
I20240502 21:27:39.250833 6514 leader_election.cc:302] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2; no voters:
I20240502 21:27:39.251227 9126 raft_consensus.cc:2781] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:27:39.251185 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093f8fd73ec34d8b9954c1716e910a78" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.251008 6432 catalog_manager.cc:5484] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 3 to 5, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 5 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.251358 6841 raft_consensus.cc:3032] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:39.251447 6841 raft_consensus.cc:729] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:39.251477 9126 raft_consensus.cc:686] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 4 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.251678 6841 consensus_queue.cc:260] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.251710 9126 consensus_queue.cc:237] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.251945 6841 raft_consensus.cc:3037] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:39.252524 6577 leader_election.cc:288] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 4 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:39.253340 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1325bd7bbb4340f88bdc7c705934862e"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:39.253578 6577 raft_consensus.cc:482] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.253717 6577 raft_consensus.cc:3037] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.255769 6841 raft_consensus.cc:2396] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:27:39.256446 6577 raft_consensus.cc:504] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.256306 6432 catalog_manager.cc:5484] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 3 to 4, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 4 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.257437 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1325bd7bbb4340f88bdc7c705934862e" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:39.257659 6711 raft_consensus.cc:3037] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.260167 6711 raft_consensus.cc:2454] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 5.
I20240502 21:27:39.260592 6515 leader_election.cc:302] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:39.260687 6577 leader_election.cc:288] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 5 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:39.260921 9126 raft_consensus.cc:2781] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:27:39.261106 9126 raft_consensus.cc:686] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.261610 9126 consensus_queue.cc:237] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.261857 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1325bd7bbb4340f88bdc7c705934862e" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:39.262018 6969 raft_consensus.cc:3037] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.264279 6969 raft_consensus.cc:2396] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:27:39.264571 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "875f83053fee45b3bc1921b24745e9da"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:39.264777 6577 raft_consensus.cc:482] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.264889 6577 raft_consensus.cc:3037] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:39.265977 6432 catalog_manager.cc:5484] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 3 to 5, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 5 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.267505 6577 raft_consensus.cc:504] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.268275 6577 leader_election.cc:288] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 4 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:39.268651 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875f83053fee45b3bc1921b24745e9da" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:39.268806 6969 raft_consensus.cc:3037] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:39.268760 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875f83053fee45b3bc1921b24745e9da" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:39.268977 6711 raft_consensus.cc:3032] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:39.269138 6711 raft_consensus.cc:729] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:39.269402 6711 consensus_queue.cc:260] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.269770 6711 raft_consensus.cc:3037] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:39.271245 6969 raft_consensus.cc:2454] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 4.
I20240502 21:27:39.271996 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "533d0913d0bc4582b976a58bda8176bc"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:39.272157 6577 raft_consensus.cc:482] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.272147 6711 raft_consensus.cc:2454] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 4.
I20240502 21:27:39.272303 6577 raft_consensus.cc:3037] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.272501 6515 leader_election.cc:302] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:39.273066 9126 raft_consensus.cc:2781] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:27:39.273494 9126 raft_consensus.cc:686] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 4 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.273762 9126 consensus_queue.cc:237] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.275210 6577 raft_consensus.cc:504] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.276046 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533d0913d0bc4582b976a58bda8176bc" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:39.276245 6711 raft_consensus.cc:3037] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.276082 6432 catalog_manager.cc:5484] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 3 to 4, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 4 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.279009 6711 raft_consensus.cc:2454] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 5.
I20240502 21:27:39.279343 6515 leader_election.cc:302] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:39.279603 9126 raft_consensus.cc:2781] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:27:39.279735 9126 raft_consensus.cc:686] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.279920 9126 consensus_queue.cc:237] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.280830 6577 leader_election.cc:288] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 5 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:39.281280 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533d0913d0bc4582b976a58bda8176bc" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:39.281466 6969 raft_consensus.cc:3037] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.281728 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c9e728880af44303a975a63a4bd7bc62"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:39.281873 6577 raft_consensus.cc:482] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.281944 6577 raft_consensus.cc:3037] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.283035 6969 raft_consensus.cc:2396] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:27:39.284137 6577 raft_consensus.cc:504] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:39.284929 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e728880af44303a975a63a4bd7bc62" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:39.285105 6711 raft_consensus.cc:3037] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.287544 6711 raft_consensus.cc:2454] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 5.
I20240502 21:27:39.287904 6515 leader_election.cc:302] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:39.288203 9124 raft_consensus.cc:2781] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:27:39.288395 6577 leader_election.cc:288] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 5 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:39.289054 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e728880af44303a975a63a4bd7bc62" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:39.288415 9124 raft_consensus.cc:686] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.289319 6969 raft_consensus.cc:3037] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.289515 9124 consensus_queue.cc:237] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:39.292369 6969 raft_consensus.cc:2396] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:27:39.292641 6432 catalog_manager.cc:5484] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 3 to 5, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 5 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.296911 6969 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3093485f30b41939a1380ab7aa9e38b"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:39.297122 6969 raft_consensus.cc:482] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.297219 6969 raft_consensus.cc:3037] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.297627 6432 catalog_manager.cc:5484] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 3 to 5, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 5 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:39.299752 6969 raft_consensus.cc:504] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.301002 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3093485f30b41939a1380ab7aa9e38b" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:39.301221 6711 raft_consensus.cc:3037] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.303849 6711 raft_consensus.cc:2396] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:27:39.304354 6969 leader_election.cc:288] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 5 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:39.305141 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3093485f30b41939a1380ab7aa9e38b" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.305279 6968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b2b3e30b94d47dd822539cb30f36f81"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:39.305413 6968 raft_consensus.cc:482] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.305393 6841 raft_consensus.cc:3037] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.305497 6968 raft_consensus.cc:3037] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.307778 6968 raft_consensus.cc:504] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.308526 6968 leader_election.cc:288] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 5 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:39.309306 6970 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aae0bc634aa7424e935c8bc81a85099b"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:39.309471 6970 raft_consensus.cc:482] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.309556 6970 raft_consensus.cc:3037] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.309672 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2b3e30b94d47dd822539cb30f36f81" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.309880 6840 raft_consensus.cc:3037] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.312024 6970 raft_consensus.cc:504] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.313386 6970 leader_election.cc:288] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 6 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:39.313773 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae0bc634aa7424e935c8bc81a85099b" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:39.314002 6709 raft_consensus.cc:3037] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.314270 6970 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3619e8a307de4583a4e470ad583fd7bc"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:39.314443 6970 raft_consensus.cc:482] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.314586 6970 raft_consensus.cc:3037] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.317060 6970 raft_consensus.cc:504] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:39.317198 6709 raft_consensus.cc:2454] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 6.
I20240502 21:27:39.317633 6906 leader_election.cc:302] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:39.317893 9104 raft_consensus.cc:2781] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:27:39.318001 9104 raft_consensus.cc:686] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 6 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.318145 6841 raft_consensus.cc:2454] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 5.
I20240502 21:27:39.318284 9104 consensus_queue.cc:237] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.318583 6840 raft_consensus.cc:2454] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 5.
I20240502 21:27:39.319185 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2b3e30b94d47dd822539cb30f36f81" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:39.319408 6711 raft_consensus.cc:3037] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.319737 6904 leader_election.cc:302] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters: ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:39.320106 6904 leader_election.cc:302] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:39.320326 9002 raft_consensus.cc:2781] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:27:39.320574 9106 raft_consensus.cc:2781] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:27:39.322448 6711 raft_consensus.cc:2396] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:27:39.323798 6970 leader_election.cc:288] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 6 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:39.324141 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae0bc634aa7424e935c8bc81a85099b" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:39.324395 6577 raft_consensus.cc:3032] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:27:39.324524 6577 raft_consensus.cc:729] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.324834 6577 consensus_queue.cc:260] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.325165 6577 raft_consensus.cc:3037] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.325228 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3619e8a307de4583a4e470ad583fd7bc" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.325395 6841 raft_consensus.cc:3037] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.327706 6841 raft_consensus.cc:2454] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 6.
I20240502 21:27:39.328303 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3619e8a307de4583a4e470ad583fd7bc" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:39.328516 6580 raft_consensus.cc:3032] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:27:39.328647 6580 raft_consensus.cc:729] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.328917 6580 consensus_queue.cc:260] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:39.329173 6580 raft_consensus.cc:3037] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.329970 6577 raft_consensus.cc:2396] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:27:39.330715 9106 raft_consensus.cc:686] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.331044 9106 consensus_queue.cc:237] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.331785 6580 raft_consensus.cc:2396] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:27:39.332091 9002 raft_consensus.cc:686] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 5 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.332480 9002 consensus_queue.cc:237] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.335165 6904 leader_election.cc:302] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters: 51bc1a0939fe49348a18d3da18f9feae
I20240502 21:27:39.335678 6432 catalog_manager.cc:5484] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 5 to 6, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 6 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:39.335860 9104 raft_consensus.cc:2781] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:27:39.336033 9104 raft_consensus.cc:686] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 6 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.336279 9104 consensus_queue.cc:237] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:39.338471 6970 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3b6b526ec34c41aea64a2a19f0208f5d"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:39.339509 6970 raft_consensus.cc:482] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.339643 6970 raft_consensus.cc:3037] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.342686 6970 raft_consensus.cc:504] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.343444 6970 leader_election.cc:288] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 5 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:39.343974 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6b526ec34c41aea64a2a19f0208f5d" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:39.344152 6580 raft_consensus.cc:3037] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.345728 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6b526ec34c41aea64a2a19f0208f5d" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.345868 6841 raft_consensus.cc:3037] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.346438 6580 raft_consensus.cc:2454] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 5.
I20240502 21:27:39.346868 6904 leader_election.cc:302] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2; no voters:
I20240502 21:27:39.347174 9158 raft_consensus.cc:2781] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:27:39.347297 9158 raft_consensus.cc:686] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 5 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.347535 9158 consensus_queue.cc:237] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.347983 6841 raft_consensus.cc:2396] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:27:39.347764 6432 catalog_manager.cc:5484] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 3 to 5, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 5 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.349239 6432 catalog_manager.cc:5484] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 5 to 6, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 6 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.349485 6970 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c10bca264dda4180bd19beb5767bf047"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:39.349643 6970 raft_consensus.cc:482] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.349787 6970 raft_consensus.cc:3037] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.349898 6432 catalog_manager.cc:5484] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 3 to 5, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 5 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:39.352713 6970 raft_consensus.cc:504] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.353500 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10bca264dda4180bd19beb5767bf047" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.353595 6970 leader_election.cc:288] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 5 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:39.353713 6841 raft_consensus.cc:3037] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.355378 6970 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "66198d766e6e4ceb820bc92bcca67d85"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:39.355540 6970 raft_consensus.cc:482] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.355643 6970 raft_consensus.cc:3037] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.355649 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10bca264dda4180bd19beb5767bf047" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:39.355857 6711 raft_consensus.cc:3037] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.357347 6841 raft_consensus.cc:2454] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 5.
I20240502 21:27:39.358095 6970 raft_consensus.cc:504] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:39.358038 6432 catalog_manager.cc:5484] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 3 to 5, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 5 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:39.358199 6904 leader_election.cc:302] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:39.358554 6711 raft_consensus.cc:2396] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:27:39.359000 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66198d766e6e4ceb820bc92bcca67d85" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.359037 9158 raft_consensus.cc:2781] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:27:39.359169 6841 raft_consensus.cc:3037] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.359470 6970 leader_election.cc:288] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 6 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:39.359987 9158 raft_consensus.cc:686] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.360231 9158 consensus_queue.cc:237] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.361111 6970 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "331bcde3e6bf43c081af836c8cdebdc7"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:39.361263 6970 raft_consensus.cc:482] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.361385 6970 raft_consensus.cc:3037] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.361969 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66198d766e6e4ceb820bc92bcca67d85" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:39.362176 6580 raft_consensus.cc:3032] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:27:39.362176 6841 raft_consensus.cc:2454] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 6.
I20240502 21:27:39.362246 6580 raft_consensus.cc:729] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.362512 6904 leader_election.cc:302] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:39.362511 6580 consensus_queue.cc:260] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:39.362723 6580 raft_consensus.cc:3037] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.362793 9158 raft_consensus.cc:2781] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:27:39.362917 9158 raft_consensus.cc:686] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.363152 9158 consensus_queue.cc:237] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:39.364692 6970 raft_consensus.cc:504] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.365803 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331bcde3e6bf43c081af836c8cdebdc7" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:39.366019 6577 raft_consensus.cc:3032] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 4 LEADER]: Stepping down as leader of term 4
I20240502 21:27:39.366132 6577 raft_consensus.cc:729] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.366417 6577 consensus_queue.cc:260] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.366792 6577 raft_consensus.cc:3037] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.367753 6580 raft_consensus.cc:2396] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:27:39.371623 6432 catalog_manager.cc:5484] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 3 to 5, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 5 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.371959 6432 catalog_manager.cc:5484] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 5 to 6, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 6 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.372223 6577 raft_consensus.cc:2396] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:27:39.372646 6970 leader_election.cc:288] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 5 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:39.373375 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331bcde3e6bf43c081af836c8cdebdc7" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:39.373571 6711 raft_consensus.cc:3037] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.375525 6969 raft_consensus.cc:1261] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Refusing update from remote peer 9793130a8e4f45388357b8d2e9f47720: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 21:27:39.376173 8281 consensus_queue.cc:1035] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:39.377007 6711 raft_consensus.cc:2454] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 5.
I20240502 21:27:39.377331 6970 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57e37f7e15b74819b692792eaed94f24"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:39.377555 6906 leader_election.cc:302] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 51bc1a0939fe49348a18d3da18f9feae
I20240502 21:27:39.377581 6970 raft_consensus.cc:482] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.378204 6970 raft_consensus.cc:3037] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:39.379045 9104 raft_consensus.cc:2781] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:27:39.379235 9104 raft_consensus.cc:686] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.379551 9104 consensus_queue.cc:237] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.380745 6970 raft_consensus.cc:504] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.381461 6970 leader_election.cc:288] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 4 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:39.382171 6970 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fbec2b1b80644a93b9239600157ad957"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:39.382297 6970 raft_consensus.cc:482] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.382391 6970 raft_consensus.cc:3037] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.383419 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e37f7e15b74819b692792eaed94f24" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:39.383639 6711 raft_consensus.cc:3037] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:39.384387 6970 raft_consensus.cc:504] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.384851 6970 leader_election.cc:288] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 5 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:39.385604 6970 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ea3b52c673ae4b9f931059c4b0a456fa"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:39.385735 6970 raft_consensus.cc:482] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.385797 6970 raft_consensus.cc:3037] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.387162 6711 raft_consensus.cc:2454] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 4.
I20240502 21:27:39.387574 6906 leader_election.cc:302] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:39.387797 9002 raft_consensus.cc:2781] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:27:39.387789 6709 raft_consensus.cc:1261] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Refusing update from remote peer 9793130a8e4f45388357b8d2e9f47720: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 21:27:39.387948 6709 pending_rounds.cc:76] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef: Aborting all ops after (but not including) 3
I20240502 21:27:39.387960 9002 raft_consensus.cc:686] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 4 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.388015 6709 pending_rounds.cc:97] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 21:27:39.387984 6970 raft_consensus.cc:504] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.388116 6709 raft_consensus.cc:2866] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:27:39.388211 9002 consensus_queue.cc:237] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.388769 6970 leader_election.cc:288] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 5 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:39.389715 9079 consensus_queue.cc:1035] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:39.390158 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3b52c673ae4b9f931059c4b0a456fa" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.390400 6839 raft_consensus.cc:3037] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.390626 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e37f7e15b74819b692792eaed94f24" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.390864 6841 raft_consensus.cc:3032] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:39.390964 6841 raft_consensus.cc:729] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:39.391245 6432 catalog_manager.cc:5484] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 4 to 5, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 5 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.391355 6841 consensus_queue.cc:260] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.391824 6841 raft_consensus.cc:3037] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:39.393487 6839 raft_consensus.cc:2454] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 5.
I20240502 21:27:39.394850 6970 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fb023d72aa8d4248a64d4723e18a3cf6"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:39.395012 6970 raft_consensus.cc:482] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.395156 6970 raft_consensus.cc:3037] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.395624 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec2b1b80644a93b9239600157ad957" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.395855 6840 raft_consensus.cc:3037] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.397697 6970 raft_consensus.cc:504] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.398520 6970 leader_election.cc:288] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 6 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:39.398828 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb023d72aa8d4248a64d4723e18a3cf6" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:39.399004 6711 raft_consensus.cc:3037] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.399107 6840 raft_consensus.cc:2396] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:27:39.399257 6970 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "980f9cc0cdc6452d8009e9f75a0d4275"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:39.399384 6970 raft_consensus.cc:482] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.399444 6970 raft_consensus.cc:3037] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.399502 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec2b1b80644a93b9239600157ad957" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:39.399663 6577 raft_consensus.cc:3037] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.401155 6970 raft_consensus.cc:504] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.401428 6711 raft_consensus.cc:2454] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 6.
I20240502 21:27:39.401741 6906 leader_election.cc:302] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:39.402012 9158 raft_consensus.cc:2781] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:27:39.402523 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb023d72aa8d4248a64d4723e18a3cf6" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:39.402632 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "980f9cc0cdc6452d8009e9f75a0d4275" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:39.402740 6580 raft_consensus.cc:3032] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:27:39.402778 6711 raft_consensus.cc:3037] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.402825 6580 raft_consensus.cc:729] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.403040 6577 raft_consensus.cc:2454] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 5.
I20240502 21:27:39.403106 6580 consensus_queue.cc:260] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.403383 6580 raft_consensus.cc:3037] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.403484 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "980f9cc0cdc6452d8009e9f75a0d4275" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.403632 6840 raft_consensus.cc:3037] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.403823 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3b52c673ae4b9f931059c4b0a456fa" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:39.403990 6709 raft_consensus.cc:3037] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.404965 6841 raft_consensus.cc:2454] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 4.
I20240502 21:27:39.405925 6580 raft_consensus.cc:2396] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:27:39.406387 6904 leader_election.cc:302] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2; no voters:
I20240502 21:27:39.406524 6709 raft_consensus.cc:2396] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:27:39.406831 6711 raft_consensus.cc:2396] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:27:39.406831 6840 raft_consensus.cc:2454] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 5.
I20240502 21:27:39.406997 6904 leader_election.cc:302] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:39.407259 9104 raft_consensus.cc:2781] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:27:39.407466 9104 raft_consensus.cc:686] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.407618 6904 leader_election.cc:302] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:39.407797 9104 consensus_queue.cc:237] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.407925 9159 raft_consensus.cc:2781] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:27:39.408517 9002 raft_consensus.cc:2781] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:27:39.408660 9002 raft_consensus.cc:686] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.408933 9002 consensus_queue.cc:237] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.413125 9158 raft_consensus.cc:686] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.413376 6970 leader_election.cc:288] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 5 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:39.413851 9158 consensus_queue.cc:237] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.415530 6970 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5e38d544ee14490c9fb5d8069440fd6a"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:39.415714 6970 raft_consensus.cc:482] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.415840 6970 raft_consensus.cc:3037] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.416225 9159 raft_consensus.cc:686] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 5 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.416538 9159 consensus_queue.cc:237] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.419168 6970 raft_consensus.cc:504] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.419849 6970 leader_election.cc:288] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 6 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:39.420410 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e38d544ee14490c9fb5d8069440fd6a" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.420622 6840 raft_consensus.cc:3037] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.420679 6969 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a09d2491d9dc4cbfb20f7adc1087b375"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:39.420832 6969 raft_consensus.cc:482] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.420922 6969 raft_consensus.cc:3037] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.423280 6840 raft_consensus.cc:2454] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 6.
I20240502 21:27:39.423410 6969 raft_consensus.cc:504] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.423982 6969 leader_election.cc:288] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 5 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:39.424413 6904 leader_election.cc:302] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:39.424746 9170 raft_consensus.cc:2781] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:27:39.424875 6968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5b2e443bad344ce89ce69871406c5a29"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:39.425062 6968 raft_consensus.cc:482] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.425153 6968 raft_consensus.cc:3037] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.425314 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09d2491d9dc4cbfb20f7adc1087b375" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.425462 6840 raft_consensus.cc:3037] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.424894 9170 raft_consensus.cc:686] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 6 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.426101 9170 consensus_queue.cc:237] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.427667 6968 raft_consensus.cc:504] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.427924 6840 raft_consensus.cc:2454] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 5.
I20240502 21:27:39.428269 6968 leader_election.cc:288] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 6 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:39.428704 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2e443bad344ce89ce69871406c5a29" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:39.428905 6577 raft_consensus.cc:3032] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:27:39.428985 6577 raft_consensus.cc:729] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.429270 6968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "093f8fd73ec34d8b9954c1716e910a78"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:39.429231 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09d2491d9dc4cbfb20f7adc1087b375" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:39.429244 6577 consensus_queue.cc:260] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.429422 6968 raft_consensus.cc:482] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.429482 6711 raft_consensus.cc:3037] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.429519 6968 raft_consensus.cc:3037] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.429514 6577 raft_consensus.cc:3037] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.431846 6968 raft_consensus.cc:504] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.432144 6577 raft_consensus.cc:2396] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:27:39.432415 6968 leader_election.cc:288] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 5 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:39.432654 6904 leader_election.cc:302] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:39.433965 6969 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1325bd7bbb4340f88bdc7c705934862e"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:39.434126 6969 raft_consensus.cc:482] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.434237 6969 raft_consensus.cc:3037] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.434280 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093f8fd73ec34d8b9954c1716e910a78" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.434450 6841 raft_consensus.cc:3037] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.436656 6969 raft_consensus.cc:504] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.436822 6711 raft_consensus.cc:2396] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:27:39.437381 6841 raft_consensus.cc:2396] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:27:39.437847 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e38d544ee14490c9fb5d8069440fd6a" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:39.437986 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1325bd7bbb4340f88bdc7c705934862e" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:39.438133 6580 raft_consensus.cc:3032] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:27:39.438149 6709 raft_consensus.cc:3037] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.438218 6580 raft_consensus.cc:729] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.438465 6580 consensus_queue.cc:260] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.438846 6580 raft_consensus.cc:3037] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.440622 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1325bd7bbb4340f88bdc7c705934862e" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:39.440793 6579 raft_consensus.cc:3032] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:27:39.440856 6579 raft_consensus.cc:729] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.441082 6579 consensus_queue.cc:260] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.441356 6579 raft_consensus.cc:3037] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.441709 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093f8fd73ec34d8b9954c1716e910a78" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:39.441933 6577 raft_consensus.cc:3032] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 4 LEADER]: Stepping down as leader of term 4
I20240502 21:27:39.442000 6577 raft_consensus.cc:729] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.442227 6577 consensus_queue.cc:260] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.442476 6577 raft_consensus.cc:3037] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.444016 6579 raft_consensus.cc:2396] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:27:39.444587 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2e443bad344ce89ce69871406c5a29" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.444729 6709 raft_consensus.cc:2454] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 6.
I20240502 21:27:39.444773 6840 raft_consensus.cc:3037] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.445134 6906 leader_election.cc:302] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:39.437361 6969 leader_election.cc:288] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 6 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:39.445477 9104 raft_consensus.cc:2781] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:27:39.445657 9104 raft_consensus.cc:686] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.445938 9104 consensus_queue.cc:237] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.446764 9106 raft_consensus.cc:2781] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:27:39.446986 6840 raft_consensus.cc:2454] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 6.
I20240502 21:27:39.448041 6577 raft_consensus.cc:2396] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:27:39.448560 6580 raft_consensus.cc:2396] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:27:39.455394 6904 leader_election.cc:302] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2; no voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720
I20240502 21:27:39.455961 6904 leader_election.cc:302] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters: 51bc1a0939fe49348a18d3da18f9feae
I20240502 21:27:39.456223 9002 raft_consensus.cc:2726] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240502 21:27:39.456394 9002 raft_consensus.cc:2781] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:27:39.456578 9002 raft_consensus.cc:686] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.456826 9002 consensus_queue.cc:237] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.456676 6432 catalog_manager.cc:5484] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 5 to 6, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 6 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:39.457230 6432 catalog_manager.cc:5484] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 3 to 4, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 4 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.464629 6968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "875f83053fee45b3bc1921b24745e9da"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:39.464892 6968 raft_consensus.cc:482] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.465097 6968 raft_consensus.cc:3037] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.472504 9106 raft_consensus.cc:686] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 6 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.472754 9106 consensus_queue.cc:237] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.483036 6968 raft_consensus.cc:504] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.483785 6968 leader_election.cc:288] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 5 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:39.485186 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875f83053fee45b3bc1921b24745e9da" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:39.485389 6709 raft_consensus.cc:3037] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.487696 6709 raft_consensus.cc:2454] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 5.
I20240502 21:27:39.488051 6906 leader_election.cc:302] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:39.488647 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875f83053fee45b3bc1921b24745e9da" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:39.488873 6580 raft_consensus.cc:3032] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 4 LEADER]: Stepping down as leader of term 4
I20240502 21:27:39.488951 6580 raft_consensus.cc:729] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.489182 6580 consensus_queue.cc:260] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.489626 6580 raft_consensus.cc:3037] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.489818 6432 catalog_manager.cc:5484] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 3 to 5, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 5 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.490343 6432 catalog_manager.cc:5484] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 5 to 6, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 6 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:39.490803 6968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "444ccd95dc0f4bdd806ca9151d56d5cd"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:39.490964 6968 raft_consensus.cc:482] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.490816 6432 catalog_manager.cc:5484] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 3 to 5, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 5 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.491080 6968 raft_consensus.cc:3037] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:39.491281 6432 catalog_manager.cc:5484] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 5 to 6, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 6 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:39.491734 6432 catalog_manager.cc:5484] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 5 to 6, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 6 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:39.492131 6580 raft_consensus.cc:2396] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:27:39.492151 6432 catalog_manager.cc:5484] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 3 to 5, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 5 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.492682 6432 catalog_manager.cc:5484] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 3 to 5, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 5 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.493822 6709 raft_consensus.cc:1261] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Refusing update from remote peer 51bc1a0939fe49348a18d3da18f9feae: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240502 21:27:39.494351 9106 raft_consensus.cc:2781] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:27:39.494616 9106 raft_consensus.cc:686] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 5 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.494626 9126 consensus_queue.cc:1035] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:39.494972 9106 consensus_queue.cc:237] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.499147 6968 raft_consensus.cc:504] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.501796 6968 leader_election.cc:288] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 4 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:39.501822 6840 raft_consensus.cc:1261] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Refusing update from remote peer 51bc1a0939fe49348a18d3da18f9feae: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20240502 21:27:39.501922 6840 pending_rounds.cc:76] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720: Aborting all ops after (but not including) 4
I20240502 21:27:39.502022 6840 pending_rounds.cc:97] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20240502 21:27:39.502084 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444ccd95dc0f4bdd806ca9151d56d5cd" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.502178 6840 raft_consensus.cc:2866] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:27:39.502240 6841 raft_consensus.cc:3032] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:39.502329 6841 raft_consensus.cc:729] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:39.502569 6841 consensus_queue.cc:260] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.502995 6841 raft_consensus.cc:3037] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:39.503861 8965 consensus_queue.cc:1035] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:39.501663 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444ccd95dc0f4bdd806ca9151d56d5cd" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:39.504156 6709 raft_consensus.cc:3037] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:39.505451 6841 raft_consensus.cc:2454] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 4.
I20240502 21:27:39.506484 6709 raft_consensus.cc:2454] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 4.
I20240502 21:27:39.506850 6906 leader_election.cc:302] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:39.507135 9106 raft_consensus.cc:2781] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:27:39.507277 9106 raft_consensus.cc:686] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 4 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.507557 9106 consensus_queue.cc:237] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.509486 6968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c9e728880af44303a975a63a4bd7bc62"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:39.509670 6968 raft_consensus.cc:482] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.509753 6968 raft_consensus.cc:3037] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.510108 6436 catalog_manager.cc:5484] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 4 to 5, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 5 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.511318 6968 raft_consensus.cc:504] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:39.511771 6968 leader_election.cc:288] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 6 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:39.512212 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e728880af44303a975a63a4bd7bc62" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:39.512405 6580 raft_consensus.cc:3032] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:27:39.512501 6580 raft_consensus.cc:729] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.512521 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e728880af44303a975a63a4bd7bc62" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:39.512676 6709 raft_consensus.cc:3037] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.512744 6580 consensus_queue.cc:260] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:39.513036 6580 raft_consensus.cc:3037] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.513382 6968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "533d0913d0bc4582b976a58bda8176bc"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:39.513542 6968 raft_consensus.cc:482] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.513661 6968 raft_consensus.cc:3037] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.514938 6709 raft_consensus.cc:2454] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 6.
I20240502 21:27:39.515304 6906 leader_election.cc:302] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:39.515589 9170 raft_consensus.cc:2781] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:27:39.515748 9170 raft_consensus.cc:686] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.515985 9170 consensus_queue.cc:237] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:39.516996 6580 raft_consensus.cc:2396] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:27:39.515990 6968 raft_consensus.cc:504] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.517942 6968 leader_election.cc:288] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 6 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:39.518265 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533d0913d0bc4582b976a58bda8176bc" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:39.518478 6709 raft_consensus.cc:3037] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.518522 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533d0913d0bc4582b976a58bda8176bc" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:39.518724 6580 raft_consensus.cc:3032] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:27:39.518822 6580 raft_consensus.cc:729] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.518584 6436 catalog_manager.cc:5484] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 3 to 4, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 4 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:39.519069 6580 consensus_queue.cc:260] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.519313 6580 raft_consensus.cc:3037] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.520808 6709 raft_consensus.cc:2454] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 6.
I20240502 21:27:39.521335 6906 leader_election.cc:302] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:39.521819 9170 raft_consensus.cc:2781] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:27:39.521937 9170 raft_consensus.cc:686] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 6 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.522140 9170 consensus_queue.cc:237] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.522776 6580 raft_consensus.cc:2396] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:27:39.525090 6436 catalog_manager.cc:5484] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 5 to 6, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 6 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.526664 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b2b3e30b94d47dd822539cb30f36f81"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:39.526800 6709 raft_consensus.cc:482] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.526911 6709 raft_consensus.cc:3037] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.528690 6709 raft_consensus.cc:504] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.529470 6709 leader_election.cc:288] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 6 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:39.529770 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2b3e30b94d47dd822539cb30f36f81" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.529961 6840 raft_consensus.cc:3037] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.529954 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2b3e30b94d47dd822539cb30f36f81" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:39.530130 6968 raft_consensus.cc:3032] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:27:39.530184 6968 raft_consensus.cc:729] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.530391 6968 consensus_queue.cc:260] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.530603 6968 raft_consensus.cc:3037] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.532315 6840 raft_consensus.cc:2454] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 6.
I20240502 21:27:39.532776 6644 leader_election.cc:302] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:39.532629 6436 catalog_manager.cc:5484] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 5 to 6, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 6 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:39.533020 6968 raft_consensus.cc:2396] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:27:39.533068 9027 raft_consensus.cc:2781] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:27:39.533226 9027 raft_consensus.cc:686] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:39.533526 9027 consensus_queue.cc:237] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.535020 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "df917e7196ed4939b75dc52930c87640"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:39.535288 6709 raft_consensus.cc:482] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.535388 6709 raft_consensus.cc:3037] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.536069 6432 catalog_manager.cc:5484] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 5 to 6, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 6 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.538906 6709 raft_consensus.cc:504] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.539510 6709 leader_election.cc:288] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 5 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:39.540279 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df917e7196ed4939b75dc52930c87640" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:39.540542 6580 raft_consensus.cc:3032] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 4 LEADER]: Stepping down as leader of term 4
I20240502 21:27:39.540544 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df917e7196ed4939b75dc52930c87640" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.540627 6580 raft_consensus.cc:729] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.540726 6840 raft_consensus.cc:3037] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.540859 6580 consensus_queue.cc:260] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.541163 6580 raft_consensus.cc:3037] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.543077 6840 raft_consensus.cc:2454] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 5.
I20240502 21:27:39.543582 6580 raft_consensus.cc:2454] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 5.
I20240502 21:27:39.543588 6644 leader_election.cc:302] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:39.543833 9027 raft_consensus.cc:2781] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:27:39.544016 9027 raft_consensus.cc:686] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:39.544265 9027 consensus_queue.cc:237] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.546636 6432 catalog_manager.cc:5484] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 4 to 5, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 5 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.547638 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aae0bc634aa7424e935c8bc81a85099b"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:39.547801 6709 raft_consensus.cc:482] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.547907 6709 raft_consensus.cc:3037] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:39.550334 6709 raft_consensus.cc:504] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.551396 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae0bc634aa7424e935c8bc81a85099b" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:39.551587 6580 raft_consensus.cc:3037] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:39.551584 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae0bc634aa7424e935c8bc81a85099b" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:39.551810 6968 raft_consensus.cc:3032] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 6 LEADER]: Stepping down as leader of term 6
I20240502 21:27:39.551899 6968 raft_consensus.cc:729] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.552204 6968 consensus_queue.cc:260] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.552517 6968 raft_consensus.cc:3037] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:39.554266 6580 raft_consensus.cc:2396] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:39.554605 6709 leader_election.cc:288] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 7 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:39.556322 6968 raft_consensus.cc:2396] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:39.556733 6645 leader_election.cc:302] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:39.557157 8274 raft_consensus.cc:2726] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240502 21:27:39.557466 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:39.557703 6709 raft_consensus.cc:482] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.557754 8750 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:39.557854 6709 raft_consensus.cc:3037] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.560086 6709 raft_consensus.cc:504] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.560930 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:39.561092 6580 raft_consensus.cc:3032] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:27:39.561194 6580 raft_consensus.cc:729] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.561538 6580 consensus_queue.cc:260] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.561825 6580 raft_consensus.cc:3037] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.568332 6580 raft_consensus.cc:2396] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:27:39.569195 6709 leader_election.cc:288] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 6 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:39.570528 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.570755 6840 raft_consensus.cc:3037] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.573362 6840 raft_consensus.cc:2454] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 6.
I20240502 21:27:39.573421 6558 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:39.573784 6644 leader_election.cc:302] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 51bc1a0939fe49348a18d3da18f9feae
I20240502 21:27:39.574448 8274 raft_consensus.cc:2781] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:27:39.574601 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c10bca264dda4180bd19beb5767bf047"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:39.574736 6709 raft_consensus.cc:482] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.574833 6709 raft_consensus.cc:3037] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.574996 8274 raft_consensus.cc:686] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:39.575246 8274 consensus_queue.cc:237] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.577289 6709 raft_consensus.cc:504] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.577994 6709 leader_election.cc:288] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 6 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:39.578706 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10bca264dda4180bd19beb5767bf047" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.578872 6841 raft_consensus.cc:3037] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.581426 6841 raft_consensus.cc:2454] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 6.
I20240502 21:27:39.583431 6644 leader_election.cc:302] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:39.583732 9027 raft_consensus.cc:2781] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:27:39.583863 9027 raft_consensus.cc:686] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:39.584201 9027 consensus_queue.cc:237] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.587065 6840 raft_consensus.cc:1261] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Refusing update from remote peer 51bc1a0939fe49348a18d3da18f9feae: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240502 21:27:39.587723 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "da14dcc58d684704b3649e85b26fb43c"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:39.587996 8965 consensus_queue.cc:1035] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:39.588804 6709 raft_consensus.cc:482] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.588994 6709 raft_consensus.cc:3037] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.592556 6709 raft_consensus.cc:504] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:39.602371 6709 leader_election.cc:288] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 6 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:39.593550 6711 raft_consensus.cc:1261] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Refusing update from remote peer 51bc1a0939fe49348a18d3da18f9feae: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240502 21:27:39.604928 6711 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f0b941baaab542a78748d4d0894cfbf4"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:39.605123 6711 raft_consensus.cc:482] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.605239 6711 raft_consensus.cc:3037] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.611521 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da14dcc58d684704b3649e85b26fb43c" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:39.611723 6580 raft_consensus.cc:3032] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:27:39.611835 6580 raft_consensus.cc:729] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.612171 6580 consensus_queue.cc:260] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:39.612534 6580 raft_consensus.cc:3037] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.616087 6580 raft_consensus.cc:2396] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:27:39.617123 9126 consensus_queue.cc:1035] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:39.608403 6711 raft_consensus.cc:504] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.618934 6711 leader_election.cc:288] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 6 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:39.619387 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da14dcc58d684704b3649e85b26fb43c" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.619603 6840 raft_consensus.cc:3037] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.623121 6840 raft_consensus.cc:2454] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 6.
I20240502 21:27:39.628077 6709 raft_consensus.cc:1226] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Rejecting Update request from peer 51bc1a0939fe49348a18d3da18f9feae for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20240502 21:27:39.628855 9126 consensus_queue.cc:1046] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:39.629122 9126 raft_consensus.cc:3032] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:27:39.629240 9126 raft_consensus.cc:729] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.629494 9126 consensus_queue.cc:260] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.629920 9126 raft_consensus.cc:3037] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.644623 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10bca264dda4180bd19beb5767bf047" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:39.645920 6968 raft_consensus.cc:3032] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:27:39.645956 6840 raft_consensus.cc:1261] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Refusing update from remote peer 51bc1a0939fe49348a18d3da18f9feae: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240502 21:27:39.646027 6968 raft_consensus.cc:729] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.646418 9126 consensus_queue.cc:1035] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:39.646333 6968 consensus_queue.cc:260] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.646644 6968 raft_consensus.cc:3037] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.650355 6968 raft_consensus.cc:2396] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:27:39.655908 6709 raft_consensus.cc:1261] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Refusing update from remote peer 51bc1a0939fe49348a18d3da18f9feae: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240502 21:27:39.657055 9126 consensus_queue.cc:1035] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:39.657285 9126 consensus_queue.cc:1225] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Peer ab5c055dc24f4944a58d7d2593ddc8ef log is divergent from this leader: its last log entry 3.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240502 21:27:39.658186 6711 pending_rounds.cc:76] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef: Aborting all ops after (but not including) 3
I20240502 21:27:39.658273 6711 pending_rounds.cc:97] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20240502 21:27:39.658365 6711 raft_consensus.cc:2866] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:27:39.659399 6820 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:39.661453 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b941baaab542a78748d4d0894cfbf4" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:39.661494 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b941baaab542a78748d4d0894cfbf4" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.661738 6580 raft_consensus.cc:2396] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240502 21:27:39.661782 6840 raft_consensus.cc:3037] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.664696 6644 leader_election.cc:302] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:39.665107 6432 catalog_manager.cc:5484] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 5 to 6, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 6 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.666379 8274 raft_consensus.cc:2781] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:27:39.666612 8274 raft_consensus.cc:686] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:39.667062 6840 raft_consensus.cc:2396] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240502 21:27:39.666814 6711 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fb023d72aa8d4248a64d4723e18a3cf6"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:39.667273 6711 raft_consensus.cc:482] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.667441 6711 raft_consensus.cc:3037] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:39.671825 8485 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:39.672116 6711 raft_consensus.cc:504] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.672699 6711 leader_election.cc:288] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 7 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:39.672847 8274 consensus_queue.cc:237] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:39.673293 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb023d72aa8d4248a64d4723e18a3cf6" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:39.673532 6968 raft_consensus.cc:3032] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 LEADER]: Stepping down as leader of term 6
I20240502 21:27:39.673614 6968 raft_consensus.cc:729] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.667451 6644 leader_election.cc:302] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720
I20240502 21:27:39.673871 6968 consensus_queue.cc:260] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.674049 9027 raft_consensus.cc:2726] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240502 21:27:39.674141 6968 raft_consensus.cc:3037] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:39.674887 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb023d72aa8d4248a64d4723e18a3cf6" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:39.675287 6580 raft_consensus.cc:3037] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:39.675731 6840 raft_consensus.cc:1261] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Refusing update from remote peer 51bc1a0939fe49348a18d3da18f9feae: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240502 21:27:39.675769 6711 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "980f9cc0cdc6452d8009e9f75a0d4275"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:39.675904 6711 raft_consensus.cc:482] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.676000 6711 raft_consensus.cc:3037] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.676340 9126 consensus_queue.cc:1035] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:39.676720 6949 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 21:27:39.678118 6968 raft_consensus.cc:2396] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:39.678840 6711 raft_consensus.cc:504] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.679436 6711 leader_election.cc:288] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 6 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:39.680310 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "980f9cc0cdc6452d8009e9f75a0d4275" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.680413 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "980f9cc0cdc6452d8009e9f75a0d4275" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:39.680505 6841 raft_consensus.cc:3037] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.680598 6969 raft_consensus.cc:3032] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:27:39.680657 6969 raft_consensus.cc:729] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.680895 6969 consensus_queue.cc:260] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.681180 6969 raft_consensus.cc:3037] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.683106 6841 raft_consensus.cc:2454] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 6.
I20240502 21:27:39.683633 6969 raft_consensus.cc:2396] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:27:39.684060 6644 leader_election.cc:302] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:39.684312 9027 raft_consensus.cc:2781] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:27:39.684448 9027 raft_consensus.cc:686] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:39.684684 9027 consensus_queue.cc:237] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.689675 6841 raft_consensus.cc:1261] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Refusing update from remote peer 51bc1a0939fe49348a18d3da18f9feae: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 21:27:39.690822 6580 raft_consensus.cc:2396] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:39.697227 9126 consensus_queue.cc:1035] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:39.698392 6644 leader_election.cc:302] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:39.699678 9025 raft_consensus.cc:2726] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240502 21:27:39.700711 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57e37f7e15b74819b692792eaed94f24"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:39.700860 6709 raft_consensus.cc:482] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.700960 6709 raft_consensus.cc:3037] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.703161 8618 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:39.703372 6709 raft_consensus.cc:504] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.703946 6709 leader_election.cc:288] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 5 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:39.704739 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e37f7e15b74819b692792eaed94f24" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:39.704975 6968 raft_consensus.cc:3032] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 21:27:39.705049 6968 raft_consensus.cc:729] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.705305 6968 consensus_queue.cc:260] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.705576 6968 raft_consensus.cc:3037] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.705896 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e37f7e15b74819b692792eaed94f24" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.706116 6840 raft_consensus.cc:3037] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.708128 6968 raft_consensus.cc:2396] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:27:39.709636 6840 raft_consensus.cc:2454] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 5.
I20240502 21:27:39.710470 6644 leader_election.cc:302] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:39.710750 9027 raft_consensus.cc:2781] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:27:39.710990 9027 raft_consensus.cc:686] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:39.711304 9027 consensus_queue.cc:237] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.712600 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "331bcde3e6bf43c081af836c8cdebdc7"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:39.712772 6709 raft_consensus.cc:482] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.712849 6709 raft_consensus.cc:3037] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.715255 6709 raft_consensus.cc:504] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.716053 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331bcde3e6bf43c081af836c8cdebdc7" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:39.716228 6968 raft_consensus.cc:3032] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:27:39.716315 6968 raft_consensus.cc:729] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.716574 6968 consensus_queue.cc:260] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.716833 6968 raft_consensus.cc:3037] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.717656 6709 leader_election.cc:288] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 6 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:39.719110 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331bcde3e6bf43c081af836c8cdebdc7" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:39.719425 6968 raft_consensus.cc:2396] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:27:39.719584 6580 raft_consensus.cc:3037] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.719825 6711 raft_consensus.cc:1261] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Refusing update from remote peer 51bc1a0939fe49348a18d3da18f9feae: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240502 21:27:39.722456 9126 consensus_queue.cc:1035] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:39.722599 9126 consensus_queue.cc:1225] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Peer ab5c055dc24f4944a58d7d2593ddc8ef log is divergent from this leader: its last log entry 3.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240502 21:27:39.723392 6709 pending_rounds.cc:76] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef: Aborting all ops after (but not including) 3
I20240502 21:27:39.723502 6709 pending_rounds.cc:97] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20240502 21:27:39.723608 6709 raft_consensus.cc:2866] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:27:39.724234 6580 raft_consensus.cc:2396] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:27:39.724726 6644 leader_election.cc:302] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:39.725241 9027 raft_consensus.cc:2726] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240502 21:27:39.725234 6710 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c6d11cba6ca04c17a9f47506824d9aa2"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:39.726555 6436 catalog_manager.cc:5484] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 5 to 6, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 6 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:39.727094 6436 catalog_manager.cc:5484] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 5 to 6, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 6 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.735953 6710 raft_consensus.cc:482] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.736061 6710 raft_consensus.cc:3037] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.738602 6710 raft_consensus.cc:504] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.739720 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d11cba6ca04c17a9f47506824d9aa2" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.739742 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d11cba6ca04c17a9f47506824d9aa2" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:39.739888 6839 raft_consensus.cc:3037] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.739888 6580 raft_consensus.cc:3032] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:27:39.739970 6580 raft_consensus.cc:729] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.740175 6580 consensus_queue.cc:260] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.740499 6580 raft_consensus.cc:3037] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.742509 6839 raft_consensus.cc:2454] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 6.
I20240502 21:27:39.743032 6710 leader_election.cc:288] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 6 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:39.743830 6580 raft_consensus.cc:2454] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 6.
I20240502 21:27:39.744256 6840 raft_consensus.cc:1261] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Refusing update from remote peer 80a838c00b544bf68ec4cc9c513212b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240502 21:27:39.744839 9170 consensus_queue.cc:1035] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:39.746908 6644 leader_election.cc:302] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:39.747138 9027 raft_consensus.cc:2781] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:27:39.747287 9027 raft_consensus.cc:686] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:39.747579 9027 consensus_queue.cc:237] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.748091 6436 catalog_manager.cc:5484] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 5 to 6, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 6 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:39.748651 6436 catalog_manager.cc:5484] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 4 to 5, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 5 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.753953 6690 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:27:39.755091 6436 catalog_manager.cc:5484] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 5 to 6, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 6 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:39.755441 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:39.757299 6580 raft_consensus.cc:1261] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Refusing update from remote peer 80a838c00b544bf68ec4cc9c513212b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240502 21:27:39.757938 9170 consensus_queue.cc:1035] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:39.763280 6710 raft_consensus.cc:1261] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Refusing update from remote peer 51bc1a0939fe49348a18d3da18f9feae: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 21:27:39.763742 8965 consensus_queue.cc:1035] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:39.780568 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "444ccd95dc0f4bdd806ca9151d56d5cd"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:39.780742 6709 raft_consensus.cc:482] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.780828 6709 raft_consensus.cc:3037] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.783215 6709 raft_consensus.cc:504] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.783668 6841 raft_consensus.cc:1261] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Refusing update from remote peer 80a838c00b544bf68ec4cc9c513212b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240502 21:27:39.784134 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444ccd95dc0f4bdd806ca9151d56d5cd" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.784301 6841 raft_consensus.cc:3037] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.784461 9178 consensus_queue.cc:1035] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:39.784632 9178 consensus_queue.cc:1225] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Peer 9793130a8e4f45388357b8d2e9f47720 log is divergent from this leader: its last log entry 3.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240502 21:27:39.786640 6841 raft_consensus.cc:2454] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 5.
I20240502 21:27:39.786859 6839 pending_rounds.cc:76] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720: Aborting all ops after (but not including) 3
I20240502 21:27:39.786943 6839 pending_rounds.cc:97] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20240502 21:27:39.787012 6839 raft_consensus.cc:2866] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:27:39.787274 6644 leader_election.cc:302] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:39.787758 9025 raft_consensus.cc:2781] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:27:39.787889 9025 raft_consensus.cc:686] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:39.788122 9025 consensus_queue.cc:237] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.792733 6709 leader_election.cc:288] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 5 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:39.793365 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444ccd95dc0f4bdd806ca9151d56d5cd" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:39.793560 6968 raft_consensus.cc:3032] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 21:27:39.793620 6968 raft_consensus.cc:729] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.793825 6968 consensus_queue.cc:260] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.794034 6968 raft_consensus.cc:3037] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.797850 6432 catalog_manager.cc:5484] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 4 to 5, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 5 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.796411 6968 raft_consensus.cc:2396] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:27:39.804973 6711 raft_consensus.cc:1261] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Refusing update from remote peer 80a838c00b544bf68ec4cc9c513212b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240502 21:27:39.805761 6580 raft_consensus.cc:1261] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Refusing update from remote peer 80a838c00b544bf68ec4cc9c513212b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240502 21:27:39.806252 9002 consensus_queue.cc:1035] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:39.806421 9002 consensus_queue.cc:1225] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Peer ab5c055dc24f4944a58d7d2593ddc8ef log is divergent from this leader: its last log entry 3.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240502 21:27:39.806710 9002 consensus_queue.cc:1035] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:39.807585 6711 pending_rounds.cc:76] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef: Aborting all ops after (but not including) 3
I20240502 21:27:39.807689 6711 pending_rounds.cc:97] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20240502 21:27:39.807770 6711 raft_consensus.cc:2866] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:27:39.808302 6577 raft_consensus.cc:1261] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Refusing update from remote peer 80a838c00b544bf68ec4cc9c513212b2: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 5. (term mismatch)
I20240502 21:27:39.808480 6577 pending_rounds.cc:76] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae: Aborting all ops after (but not including) 4
I20240502 21:27:39.808588 6577 pending_rounds.cc:97] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20240502 21:27:39.808774 6577 raft_consensus.cc:2866] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:27:39.809981 9002 consensus_queue.cc:1035] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:39.810164 9002 consensus_queue.cc:1225] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Peer 51bc1a0939fe49348a18d3da18f9feae log is divergent from this leader: its last log entry 2.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240502 21:27:39.811112 6577 pending_rounds.cc:76] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae: Aborting all ops after (but not including) 3
I20240502 21:27:39.811223 6577 pending_rounds.cc:97] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 21:27:39.811298 6577 raft_consensus.cc:2866] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:27:39.813974 6840 raft_consensus.cc:1261] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Refusing update from remote peer 80a838c00b544bf68ec4cc9c513212b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240502 21:27:39.814954 9170 consensus_queue.cc:1035] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:39.819983 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3093485f30b41939a1380ab7aa9e38b"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:39.830417 6709 raft_consensus.cc:482] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.830551 6709 raft_consensus.cc:3037] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.843405 6839 raft_consensus.cc:1261] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Refusing update from remote peer 80a838c00b544bf68ec4cc9c513212b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240502 21:27:39.843899 6840 raft_consensus.cc:1261] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Refusing update from remote peer 80a838c00b544bf68ec4cc9c513212b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240502 21:27:39.844584 9106 consensus_queue.cc:1035] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:39.844755 9106 consensus_queue.cc:1225] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Peer 9793130a8e4f45388357b8d2e9f47720 log is divergent from this leader: its last log entry 3.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240502 21:27:39.844681 9104 consensus_queue.cc:1035] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:39.845587 6840 raft_consensus.cc:1261] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Refusing update from remote peer 80a838c00b544bf68ec4cc9c513212b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240502 21:27:39.846735 9104 consensus_queue.cc:1035] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:39.848940 6839 pending_rounds.cc:76] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720: Aborting all ops after (but not including) 3
I20240502 21:27:39.850023 6580 raft_consensus.cc:1261] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Refusing update from remote peer 80a838c00b544bf68ec4cc9c513212b2: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 5. (term mismatch)
I20240502 21:27:39.850108 6580 pending_rounds.cc:76] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae: Aborting all ops after (but not including) 4
I20240502 21:27:39.850167 6580 pending_rounds.cc:97] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20240502 21:27:39.850225 6580 raft_consensus.cc:2866] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:27:39.850934 9106 consensus_queue.cc:1035] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:39.851101 9106 consensus_queue.cc:1225] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Peer 51bc1a0939fe49348a18d3da18f9feae log is divergent from this leader: its last log entry 2.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240502 21:27:39.851027 6839 pending_rounds.cc:97] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20240502 21:27:39.851274 6839 raft_consensus.cc:2866] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:27:39.854564 6580 pending_rounds.cc:76] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae: Aborting all ops after (but not including) 3
I20240502 21:27:39.854683 6580 pending_rounds.cc:97] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 21:27:39.854792 6580 raft_consensus.cc:2866] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:27:39.855337 6709 raft_consensus.cc:504] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.856658 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3093485f30b41939a1380ab7aa9e38b" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:39.856863 6968 raft_consensus.cc:3032] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:27:39.856937 6968 raft_consensus.cc:729] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.857195 6968 consensus_queue.cc:260] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.857606 6968 raft_consensus.cc:3037] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.860317 6968 raft_consensus.cc:2454] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 6.
I20240502 21:27:39.860993 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3093485f30b41939a1380ab7aa9e38b" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.861081 6645 leader_election.cc:302] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:39.861176 6839 raft_consensus.cc:3037] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.864288 6840 raft_consensus.cc:1261] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Refusing update from remote peer 80a838c00b544bf68ec4cc9c513212b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240502 21:27:39.864529 9027 raft_consensus.cc:2781] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:27:39.864688 9027 raft_consensus.cc:686] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:39.864776 6839 raft_consensus.cc:2454] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 6.
I20240502 21:27:39.864789 9106 consensus_queue.cc:1035] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:39.864936 9027 consensus_queue.cc:237] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.866040 6709 leader_election.cc:288] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 6 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:39.867412 6436 catalog_manager.cc:5484] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 5 to 6, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 6 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.867669 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1c98cbfbd0a44335b75b228c217f9132"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:39.867858 6709 raft_consensus.cc:482] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.867980 6709 raft_consensus.cc:3037] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:39.872506 6711 raft_consensus.cc:1261] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Refusing update from remote peer 80a838c00b544bf68ec4cc9c513212b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240502 21:27:39.873392 9106 consensus_queue.cc:1035] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:39.873577 9106 consensus_queue.cc:1225] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Peer ab5c055dc24f4944a58d7d2593ddc8ef log is divergent from this leader: its last log entry 3.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240502 21:27:39.874435 6711 pending_rounds.cc:76] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef: Aborting all ops after (but not including) 3
I20240502 21:27:39.874531 6711 pending_rounds.cc:97] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20240502 21:27:39.874600 6711 raft_consensus.cc:2866] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:27:39.880999 6709 raft_consensus.cc:504] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:39.881405 6580 raft_consensus.cc:1261] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Refusing update from remote peer 80a838c00b544bf68ec4cc9c513212b2: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 5. (term mismatch)
I20240502 21:27:39.881528 6580 pending_rounds.cc:76] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae: Aborting all ops after (but not including) 4
I20240502 21:27:39.881594 6580 pending_rounds.cc:97] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20240502 21:27:39.881680 6580 raft_consensus.cc:2866] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:27:39.881982 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c98cbfbd0a44335b75b228c217f9132" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.882191 6841 raft_consensus.cc:3037] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:39.882236 9104 consensus_queue.cc:1035] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:39.882455 9104 consensus_queue.cc:1225] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Peer 51bc1a0939fe49348a18d3da18f9feae log is divergent from this leader: its last log entry 2.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240502 21:27:39.883878 6580 pending_rounds.cc:76] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae: Aborting all ops after (but not including) 3
I20240502 21:27:39.883970 6580 pending_rounds.cc:97] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 21:27:39.884088 6580 raft_consensus.cc:2866] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:27:39.884769 6841 raft_consensus.cc:2454] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 4.
I20240502 21:27:39.885109 6644 leader_election.cc:302] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:39.885397 9025 raft_consensus.cc:2781] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:27:39.885535 9025 raft_consensus.cc:686] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:39.885776 9025 consensus_queue.cc:237] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:39.888543 6436 catalog_manager.cc:5484] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 3 to 4, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 4 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.892634 6709 leader_election.cc:288] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 4 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:39.893779 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c98cbfbd0a44335b75b228c217f9132" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:39.893946 6580 raft_consensus.cc:3032] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:27:39.894047 6580 raft_consensus.cc:729] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.894279 6580 consensus_queue.cc:260] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:39.894676 6580 raft_consensus.cc:3037] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:27:39.896416 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ea3b52c673ae4b9f931059c4b0a456fa"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:39.896963 6709 raft_consensus.cc:482] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.897073 6709 raft_consensus.cc:3037] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.899073 6580 raft_consensus.cc:2454] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 4.
I20240502 21:27:39.901605 6709 raft_consensus.cc:504] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.902547 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3b52c673ae4b9f931059c4b0a456fa" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:39.902747 6968 raft_consensus.cc:3032] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:27:39.902814 6968 raft_consensus.cc:729] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.902937 6709 leader_election.cc:288] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 6 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:39.903033 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3b52c673ae4b9f931059c4b0a456fa" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.903067 6968 consensus_queue.cc:260] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.903234 6841 raft_consensus.cc:3037] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.903332 6968 raft_consensus.cc:3037] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.903872 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "169899da82594d9cb673f8db74764ca2"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:39.904038 6709 raft_consensus.cc:482] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.904129 6709 raft_consensus.cc:3037] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.905864 6968 raft_consensus.cc:2396] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:27:39.907200 6841 raft_consensus.cc:2454] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 6.
I20240502 21:27:39.907618 6644 leader_election.cc:302] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:39.908008 6709 raft_consensus.cc:504] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.908629 9027 raft_consensus.cc:2781] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:27:39.908694 6709 leader_election.cc:288] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 6 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:39.908800 9027 raft_consensus.cc:686] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:39.909057 9027 consensus_queue.cc:237] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.910606 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169899da82594d9cb673f8db74764ca2" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:39.910801 6580 raft_consensus.cc:3032] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:27:39.910883 6580 raft_consensus.cc:729] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:39.910964 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169899da82594d9cb673f8db74764ca2" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.911144 6839 raft_consensus.cc:3037] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.911098 6580 consensus_queue.cc:260] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.911439 6580 raft_consensus.cc:3037] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.913789 6839 raft_consensus.cc:2454] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 6.
I20240502 21:27:39.914338 6580 raft_consensus.cc:2454] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 6.
I20240502 21:27:39.914430 6644 leader_election.cc:302] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:39.914892 8274 raft_consensus.cc:2781] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:27:39.915159 8274 raft_consensus.cc:686] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:39.915383 6968 raft_consensus.cc:1261] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Refusing update from remote peer ab5c055dc24f4944a58d7d2593ddc8ef: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 5. (term mismatch)
I20240502 21:27:39.915480 6968 pending_rounds.cc:76] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2: Aborting all ops after (but not including) 4
I20240502 21:27:39.915421 8274 consensus_queue.cc:237] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.915539 6968 pending_rounds.cc:97] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20240502 21:27:39.915616 6968 raft_consensus.cc:2866] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:27:39.916755 8274 consensus_queue.cc:1035] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:39.916929 8274 consensus_queue.cc:1225] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Peer 80a838c00b544bf68ec4cc9c513212b2 log is divergent from this leader: its last log entry 2.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240502 21:27:39.917363 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a09d2491d9dc4cbfb20f7adc1087b375"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:39.917603 6709 raft_consensus.cc:482] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.917748 6709 raft_consensus.cc:3037] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.918212 6968 pending_rounds.cc:76] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2: Aborting all ops after (but not including) 3
I20240502 21:27:39.918296 6968 pending_rounds.cc:97] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 21:27:39.918368 6968 raft_consensus.cc:2866] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:27:39.918133 6432 catalog_manager.cc:5484] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 5 to 6, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 6 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.918797 6432 catalog_manager.cc:5484] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 5 to 6, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 6 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.922890 6709 raft_consensus.cc:504] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.923041 6711 raft_consensus.cc:1261] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Refusing update from remote peer 80a838c00b544bf68ec4cc9c513212b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240502 21:27:39.923593 6709 leader_election.cc:288] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 6 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:39.924331 9170 consensus_queue.cc:1035] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:39.924726 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09d2491d9dc4cbfb20f7adc1087b375" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:39.924897 6968 raft_consensus.cc:3032] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:27:39.924986 6968 raft_consensus.cc:729] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.925199 6968 consensus_queue.cc:260] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.925520 6968 raft_consensus.cc:3037] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.926673 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8fac59b25c164e658fe71cfdf7ca78cb"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:39.926833 6709 raft_consensus.cc:482] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.926913 6709 raft_consensus.cc:3037] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.927407 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09d2491d9dc4cbfb20f7adc1087b375" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.927609 6839 raft_consensus.cc:3037] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.928354 6968 raft_consensus.cc:2454] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 6.
I20240502 21:27:39.928706 6645 leader_election.cc:302] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:39.929023 9025 raft_consensus.cc:2781] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:27:39.929134 9025 raft_consensus.cc:686] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:39.929368 9025 consensus_queue.cc:237] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.930361 6839 raft_consensus.cc:2454] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 6.
I20240502 21:27:39.931267 6709 raft_consensus.cc:504] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:39.932103 6841 raft_consensus.cc:1261] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Refusing update from remote peer ab5c055dc24f4944a58d7d2593ddc8ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240502 21:27:39.932312 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fac59b25c164e658fe71cfdf7ca78cb" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:39.932600 6840 raft_consensus.cc:3037] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.932822 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fac59b25c164e658fe71cfdf7ca78cb" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:39.932984 6580 raft_consensus.cc:3037] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:39.934911 6580 raft_consensus.cc:2454] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 5.
I20240502 21:27:39.935104 6840 raft_consensus.cc:2396] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:27:39.935225 6644 leader_election.cc:302] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:39.935479 9027 raft_consensus.cc:2781] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:27:39.935590 9027 raft_consensus.cc:686] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:39.935802 9027 consensus_queue.cc:237] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:39.936913 9025 consensus_queue.cc:1035] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:39.939664 6580 raft_consensus.cc:1261] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Refusing update from remote peer 80a838c00b544bf68ec4cc9c513212b2: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 5. (term mismatch)
I20240502 21:27:39.939778 6580 pending_rounds.cc:76] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae: Aborting all ops after (but not including) 4
I20240502 21:27:39.939837 6580 pending_rounds.cc:97] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae: Aborting uncommitted NO_OP operation due to leader change: 4.5
I20240502 21:27:39.939916 6580 raft_consensus.cc:2866] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:27:39.940578 9170 consensus_queue.cc:1035] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:39.947353 6710 raft_consensus.cc:1261] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Refusing update from remote peer 80a838c00b544bf68ec4cc9c513212b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240502 21:27:39.949167 9106 consensus_queue.cc:1035] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:39.950119 6709 leader_election.cc:288] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 5 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:39.951865 6708 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1325bd7bbb4340f88bdc7c705934862e"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:39.952150 6708 raft_consensus.cc:482] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.952267 6708 raft_consensus.cc:3037] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:39.954802 6708 raft_consensus.cc:504] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.955550 6708 leader_election.cc:288] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 7 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:39.955914 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1325bd7bbb4340f88bdc7c705934862e" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:39.956137 6968 raft_consensus.cc:3032] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 6 LEADER]: Stepping down as leader of term 6
I20240502 21:27:39.956086 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1325bd7bbb4340f88bdc7c705934862e" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:39.956236 6968 raft_consensus.cc:729] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.956316 6580 raft_consensus.cc:3037] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:39.956506 6968 consensus_queue.cc:260] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.956779 6968 raft_consensus.cc:3037] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:39.957446 6708 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "875f83053fee45b3bc1921b24745e9da"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:39.957638 6708 raft_consensus.cc:482] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.957746 6708 raft_consensus.cc:3037] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.959157 6580 raft_consensus.cc:2396] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:39.959476 6968 raft_consensus.cc:2396] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:39.960155 6708 raft_consensus.cc:504] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.961431 6644 leader_election.cc:302] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:39.961877 9027 raft_consensus.cc:2726] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240502 21:27:39.962088 6577 raft_consensus.cc:1261] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Refusing update from remote peer 80a838c00b544bf68ec4cc9c513212b2: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 5. (term mismatch)
I20240502 21:27:39.962105 6708 leader_election.cc:288] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 6 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:39.962185 6577 pending_rounds.cc:76] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae: Aborting all ops after (but not including) 4
I20240502 21:27:39.962249 6577 pending_rounds.cc:97] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20240502 21:27:39.962309 6577 raft_consensus.cc:2866] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:27:39.962419 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875f83053fee45b3bc1921b24745e9da" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:39.962580 6968 raft_consensus.cc:3032] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:27:39.962662 6968 raft_consensus.cc:729] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.962929 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875f83053fee45b3bc1921b24745e9da" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:39.962952 6968 consensus_queue.cc:260] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.963101 6577 raft_consensus.cc:3037] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.963368 6968 raft_consensus.cc:3037] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.965277 6708 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "533d0913d0bc4582b976a58bda8176bc"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:39.965447 6708 raft_consensus.cc:482] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.965607 6708 raft_consensus.cc:3037] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:39.965673 9170 consensus_queue.cc:1035] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:39.965852 9170 consensus_queue.cc:1225] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Peer 51bc1a0939fe49348a18d3da18f9feae log is divergent from this leader: its last log entry 2.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240502 21:27:39.967015 6579 pending_rounds.cc:76] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae: Aborting all ops after (but not including) 3
I20240502 21:27:39.967005 6968 raft_consensus.cc:2454] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 6.
I20240502 21:27:39.967111 6579 pending_rounds.cc:97] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 21:27:39.967180 6579 raft_consensus.cc:2866] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:27:39.968812 6708 raft_consensus.cc:504] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.969027 6436 catalog_manager.cc:5484] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 3 to 5, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 5 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.969983 6577 raft_consensus.cc:2454] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 6.
I20240502 21:27:39.970208 6645 leader_election.cc:302] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:39.970487 9027 raft_consensus.cc:2781] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:27:39.969766 6436 catalog_manager.cc:5484] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 5 to 6, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 6 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:39.970595 9027 raft_consensus.cc:686] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:39.970813 9027 consensus_queue.cc:237] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:39.972481 6708 leader_election.cc:288] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 7 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:39.973083 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533d0913d0bc4582b976a58bda8176bc" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:39.973299 6577 raft_consensus.cc:3037] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:39.973758 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533d0913d0bc4582b976a58bda8176bc" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:39.973929 6968 raft_consensus.cc:3032] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 6 LEADER]: Stepping down as leader of term 6
I20240502 21:27:39.973965 6708 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c9e728880af44303a975a63a4bd7bc62"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:39.974023 6968 raft_consensus.cc:729] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.974179 6708 raft_consensus.cc:482] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.974246 6968 consensus_queue.cc:260] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.974292 6708 raft_consensus.cc:3037] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:39.974496 6968 raft_consensus.cc:3037] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:39.976518 6577 raft_consensus.cc:2396] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:39.976891 6968 raft_consensus.cc:2396] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:39.977245 6708 raft_consensus.cc:504] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:39.977308 6645 leader_election.cc:302] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:39.977818 9025 raft_consensus.cc:2726] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240502 21:27:39.978266 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e728880af44303a975a63a4bd7bc62" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:39.978510 6968 raft_consensus.cc:3032] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 LEADER]: Stepping down as leader of term 6
I20240502 21:27:39.978685 6968 raft_consensus.cc:729] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:39.979122 6708 leader_election.cc:288] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 7 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:39.979393 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e728880af44303a975a63a4bd7bc62" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:39.979583 6577 raft_consensus.cc:3037] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:39.980834 6436 catalog_manager.cc:5484] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 5 to 6, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 6 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:39.978996 6968 consensus_queue.cc:260] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:39.981269 6968 raft_consensus.cc:3037] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:39.982482 6577 raft_consensus.cc:2454] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 7.
I20240502 21:27:39.984191 6968 raft_consensus.cc:2454] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 7.
I20240502 21:27:39.984675 6645 leader_election.cc:302] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:39.984920 9025 raft_consensus.cc:2781] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Leader election won for term 7
I20240502 21:27:39.985045 9025 raft_consensus.cc:686] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:39.985261 9025 consensus_queue.cc:237] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:39.987758 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b2b3e30b94d47dd822539cb30f36f81"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:39.987998 6839 raft_consensus.cc:482] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.988106 6839 raft_consensus.cc:3037] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:39.989560 6577 raft_consensus.cc:1261] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Refusing update from remote peer ab5c055dc24f4944a58d7d2593ddc8ef: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240502 21:27:39.989691 6439 catalog_manager.cc:5484] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 6 to 7, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 7 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:39.990446 6841 raft_consensus.cc:1261] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Refusing update from remote peer ab5c055dc24f4944a58d7d2593ddc8ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240502 21:27:39.990603 6839 raft_consensus.cc:504] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.991132 9027 consensus_queue.cc:1035] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:39.993065 6839 leader_election.cc:288] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 7 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:39.994345 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2b3e30b94d47dd822539cb30f36f81" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:39.994387 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "df917e7196ed4939b75dc52930c87640"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:39.994496 6968 raft_consensus.cc:3037] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:39.994553 6839 raft_consensus.cc:482] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:39.994680 6839 raft_consensus.cc:3037] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:39.996865 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2b3e30b94d47dd822539cb30f36f81" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:39.996905 6968 raft_consensus.cc:2454] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 7.
I20240502 21:27:39.997308 6708 raft_consensus.cc:3032] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 LEADER]: Stepping down as leader of term 6
I20240502 21:27:39.997617 6708 raft_consensus.cc:729] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:39.997812 6839 raft_consensus.cc:504] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:39.998312 6708 consensus_queue.cc:260] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.999262 6775 leader_election.cc:302] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:39.999394 6708 raft_consensus.cc:3037] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:39.999537 9074 raft_consensus.cc:2781] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Leader election won for term 7
I20240502 21:27:39.999675 9074 raft_consensus.cc:686] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 7 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:39.999888 9074 consensus_queue.cc:237] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:39.998992 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df917e7196ed4939b75dc52930c87640" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.001032 6711 raft_consensus.cc:3032] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:27:40.001103 6711 raft_consensus.cc:729] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:40.001302 6711 consensus_queue.cc:260] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.001540 6711 raft_consensus.cc:3037] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:40.001735 6708 raft_consensus.cc:2454] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 7.
I20240502 21:27:40.004544 6711 raft_consensus.cc:2396] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240502 21:27:40.006503 6839 leader_election.cc:288] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 6 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:40.007612 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3619e8a307de4583a4e470ad583fd7bc"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:40.007948 6839 raft_consensus.cc:482] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.008141 6839 raft_consensus.cc:3037] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.011490 6839 raft_consensus.cc:504] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.012699 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3619e8a307de4583a4e470ad583fd7bc" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.012893 6968 raft_consensus.cc:3032] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 6 LEADER]: Stepping down as leader of term 6
I20240502 21:27:40.012964 6968 raft_consensus.cc:729] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:40.013080 6839 leader_election.cc:288] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 7 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:40.013203 6968 consensus_queue.cc:260] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.013361 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df917e7196ed4939b75dc52930c87640" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.013507 6577 raft_consensus.cc:3037] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:40.013521 6968 raft_consensus.cc:3037] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.013938 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:40.014247 6839 raft_consensus.cc:482] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.014429 6839 raft_consensus.cc:3037] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.015156 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3619e8a307de4583a4e470ad583fd7bc" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.015339 6580 raft_consensus.cc:3037] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.015822 6968 raft_consensus.cc:2454] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 7.
I20240502 21:27:40.016199 6775 leader_election.cc:302] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:40.016507 9077 raft_consensus.cc:2781] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Leader election won for term 7
I20240502 21:27:40.016616 9077 raft_consensus.cc:686] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 7 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:40.016870 9077 consensus_queue.cc:237] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.017884 6580 raft_consensus.cc:2454] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 7.
I20240502 21:27:40.019510 6839 raft_consensus.cc:504] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.020876 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.021098 6579 raft_consensus.cc:3037] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.021194 6439 catalog_manager.cc:5484] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 6 to 7, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 7 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.021736 6439 catalog_manager.cc:5484] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 6 to 7, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 7 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.023475 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.023541 6577 raft_consensus.cc:2454] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 6.
I20240502 21:27:40.023664 6708 raft_consensus.cc:3032] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 LEADER]: Stepping down as leader of term 6
I20240502 21:27:40.023734 6579 raft_consensus.cc:2396] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:40.023759 6708 raft_consensus.cc:729] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:40.024032 6708 consensus_queue.cc:260] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.024349 6708 raft_consensus.cc:3037] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.025024 6774 leader_election.cc:302] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters: ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:40.025564 9074 raft_consensus.cc:2781] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:27:40.025687 9074 raft_consensus.cc:686] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 6 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:40.025931 9074 consensus_queue.cc:237] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.028025 6708 raft_consensus.cc:2396] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:40.028498 6776 leader_election.cc:302] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720; no voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:40.028757 9077 raft_consensus.cc:2726] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240502 21:27:40.030274 6968 raft_consensus.cc:1261] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Refusing update from remote peer ab5c055dc24f4944a58d7d2593ddc8ef: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 5. (term mismatch)
I20240502 21:27:40.030385 6968 pending_rounds.cc:76] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2: Aborting all ops after (but not including) 4
I20240502 21:27:40.030439 6968 pending_rounds.cc:97] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20240502 21:27:40.030511 6968 raft_consensus.cc:2866] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:27:40.031622 8274 consensus_queue.cc:1035] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:40.031805 8274 consensus_queue.cc:1225] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Peer 80a838c00b544bf68ec4cc9c513212b2 log is divergent from this leader: its last log entry 2.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240502 21:27:40.033046 6968 pending_rounds.cc:76] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2: Aborting all ops after (but not including) 3
I20240502 21:27:40.033174 6968 pending_rounds.cc:97] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 21:27:40.033254 6968 raft_consensus.cc:2866] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:27:40.033599 6839 leader_election.cc:288] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 7 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:40.034204 6436 catalog_manager.cc:5484] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 5 to 6, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 6 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:40.035355 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c10bca264dda4180bd19beb5767bf047"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:40.035533 6839 raft_consensus.cc:482] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.035627 6839 raft_consensus.cc:3037] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.038125 6839 raft_consensus.cc:504] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.040239 6839 leader_election.cc:288] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 7 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:40.040180 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10bca264dda4180bd19beb5767bf047" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.041990 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "66198d766e6e4ceb820bc92bcca67d85"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:40.042132 6839 raft_consensus.cc:482] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.042236 6839 raft_consensus.cc:3037] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.044714 6839 raft_consensus.cc:504] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.045578 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66198d766e6e4ceb820bc92bcca67d85" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.045778 6968 raft_consensus.cc:3032] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 LEADER]: Stepping down as leader of term 6
I20240502 21:27:40.045883 6968 raft_consensus.cc:729] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:40.046058 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66198d766e6e4ceb820bc92bcca67d85" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.046149 6968 consensus_queue.cc:260] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.046312 6579 raft_consensus.cc:3037] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.046501 6968 raft_consensus.cc:3037] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.049225 6968 raft_consensus.cc:2454] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 7.
I20240502 21:27:40.049538 6579 raft_consensus.cc:2454] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 7.
I20240502 21:27:40.049856 6775 leader_election.cc:302] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:40.050108 9074 raft_consensus.cc:2781] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Leader election won for term 7
I20240502 21:27:40.050251 9074 raft_consensus.cc:686] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 7 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:40.050498 9074 consensus_queue.cc:237] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.051585 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10bca264dda4180bd19beb5767bf047" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.051764 6708 raft_consensus.cc:3032] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 LEADER]: Stepping down as leader of term 6
I20240502 21:27:40.051848 6708 raft_consensus.cc:729] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:40.052067 6708 consensus_queue.cc:260] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.052383 6708 raft_consensus.cc:3037] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.054888 6708 raft_consensus.cc:2454] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 7.
I20240502 21:27:40.055267 6776 leader_election.cc:302] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:40.057300 6579 raft_consensus.cc:1261] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Refusing update from remote peer 80a838c00b544bf68ec4cc9c513212b2: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 5. (term mismatch)
I20240502 21:27:40.057469 6579 pending_rounds.cc:76] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae: Aborting all ops after (but not including) 4
I20240502 21:27:40.057581 6579 pending_rounds.cc:97] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20240502 21:27:40.057683 6579 raft_consensus.cc:2866] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:27:40.058288 9178 consensus_queue.cc:1035] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:40.058455 9178 consensus_queue.cc:1225] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Peer 51bc1a0939fe49348a18d3da18f9feae log is divergent from this leader: its last log entry 2.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240502 21:27:40.059458 6577 pending_rounds.cc:76] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae: Aborting all ops after (but not including) 3
I20240502 21:27:40.059574 6577 pending_rounds.cc:97] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 21:27:40.059661 6577 raft_consensus.cc:2866] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:27:40.060534 9074 raft_consensus.cc:2781] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Leader election won for term 7
I20240502 21:27:40.060642 9074 raft_consensus.cc:686] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 7 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:40.060868 9074 consensus_queue.cc:237] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.064329 6840 raft_consensus.cc:1261] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Refusing update from remote peer 80a838c00b544bf68ec4cc9c513212b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240502 21:27:40.064920 9106 consensus_queue.cc:1035] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:40.075261 6439 catalog_manager.cc:5484] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 6 to 7, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 7 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.076155 6839 leader_election.cc:288] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 7 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:40.077308 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "980f9cc0cdc6452d8009e9f75a0d4275"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:40.077571 6839 raft_consensus.cc:482] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.077672 6839 raft_consensus.cc:3037] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.080911 6839 raft_consensus.cc:504] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.081434 6439 catalog_manager.cc:5484] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 6 to 7, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 7 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.082638 6839 leader_election.cc:288] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 7 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:40.083110 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "980f9cc0cdc6452d8009e9f75a0d4275" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.083348 6708 raft_consensus.cc:3032] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 LEADER]: Stepping down as leader of term 6
I20240502 21:27:40.083328 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "980f9cc0cdc6452d8009e9f75a0d4275" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.083443 6708 raft_consensus.cc:729] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:40.083514 6968 raft_consensus.cc:3037] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.084664 6708 consensus_queue.cc:260] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.085638 6708 raft_consensus.cc:3037] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.086257 6968 raft_consensus.cc:2396] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:40.086287 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f0b941baaab542a78748d4d0894cfbf4"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:40.087138 6839 raft_consensus.cc:482] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.087340 6839 raft_consensus.cc:3037] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.090950 6839 raft_consensus.cc:504] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.091814 6839 leader_election.cc:288] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 7 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:40.092186 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b941baaab542a78748d4d0894cfbf4" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.092406 6577 raft_consensus.cc:3037] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.092388 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b941baaab542a78748d4d0894cfbf4" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.092623 6711 raft_consensus.cc:3037] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.093501 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3b6b526ec34c41aea64a2a19f0208f5d"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:40.093829 6839 raft_consensus.cc:482] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.093997 6839 raft_consensus.cc:3037] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:40.095157 6577 raft_consensus.cc:2454] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 7.
I20240502 21:27:40.095187 6711 raft_consensus.cc:2454] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 7.
I20240502 21:27:40.095607 6774 leader_election.cc:302] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:40.095975 9077 raft_consensus.cc:2781] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Leader election won for term 7
I20240502 21:27:40.096127 9077 raft_consensus.cc:686] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 7 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:40.096369 9077 consensus_queue.cc:237] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.097695 6839 raft_consensus.cc:504] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.098600 6708 raft_consensus.cc:2396] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:40.098570 6839 leader_election.cc:288] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 6 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:40.099010 6776 leader_election.cc:302] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720; no voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:40.099068 6439 catalog_manager.cc:5484] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 5 to 7, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 7 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.099386 9077 raft_consensus.cc:2726] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240502 21:27:40.099601 6840 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "da14dcc58d684704b3649e85b26fb43c"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:40.099752 6840 raft_consensus.cc:482] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.099843 6840 raft_consensus.cc:3037] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.102675 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6b526ec34c41aea64a2a19f0208f5d" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.102905 6577 raft_consensus.cc:3037] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:40.103461 6840 raft_consensus.cc:504] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.104359 6840 leader_election.cc:288] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 7 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:40.104633 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da14dcc58d684704b3649e85b26fb43c" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.104789 6708 raft_consensus.cc:3032] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 LEADER]: Stepping down as leader of term 6
I20240502 21:27:40.104882 6708 raft_consensus.cc:729] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:40.105111 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da14dcc58d684704b3649e85b26fb43c" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.105122 6708 consensus_queue.cc:260] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.105306 6579 raft_consensus.cc:3037] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.105414 6708 raft_consensus.cc:3037] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.105477 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5e38d544ee14490c9fb5d8069440fd6a"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:40.105667 6839 raft_consensus.cc:482] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.105790 6839 raft_consensus.cc:3037] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.108027 6708 raft_consensus.cc:2396] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:40.108736 6579 raft_consensus.cc:2396] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:40.109138 6774 leader_election.cc:302] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720; no voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:40.109503 9077 raft_consensus.cc:2726] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240502 21:27:40.109426 6839 raft_consensus.cc:504] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.109819 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6b526ec34c41aea64a2a19f0208f5d" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.110373 6839 leader_election.cc:288] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 7 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:40.111022 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e38d544ee14490c9fb5d8069440fd6a" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.111233 6579 raft_consensus.cc:3037] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.111244 6840 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a09d2491d9dc4cbfb20f7adc1087b375"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:40.111508 6840 raft_consensus.cc:482] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.111624 6840 raft_consensus.cc:3037] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.110520 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e38d544ee14490c9fb5d8069440fd6a" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.111801 6969 raft_consensus.cc:3032] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 6 LEADER]: Stepping down as leader of term 6
I20240502 21:27:40.111894 6969 raft_consensus.cc:729] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:40.112133 6969 consensus_queue.cc:260] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.112532 6969 raft_consensus.cc:3037] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.114773 6840 raft_consensus.cc:504] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.115096 6969 raft_consensus.cc:2454] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 7.
I20240502 21:27:40.115600 6775 leader_election.cc:302] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:40.115777 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09d2491d9dc4cbfb20f7adc1087b375" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.115902 9077 raft_consensus.cc:2781] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Leader election won for term 7
I20240502 21:27:40.115955 6969 raft_consensus.cc:3037] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.116068 9077 raft_consensus.cc:686] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 7 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:40.116364 9077 consensus_queue.cc:237] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.117551 6840 leader_election.cc:288] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 7 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:40.118551 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09d2491d9dc4cbfb20f7adc1087b375" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.118600 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5b2e443bad344ce89ce69871406c5a29"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:40.118738 6708 raft_consensus.cc:3032] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 LEADER]: Stepping down as leader of term 6
I20240502 21:27:40.118851 6708 raft_consensus.cc:729] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:40.118930 6839 raft_consensus.cc:482] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.119050 6839 raft_consensus.cc:3037] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.119100 6708 consensus_queue.cc:260] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.119400 6708 raft_consensus.cc:3037] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.119709 6439 catalog_manager.cc:5484] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 6 to 7, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 7 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.121596 6708 raft_consensus.cc:2396] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240502 21:27:40.122750 6968 raft_consensus.cc:3032] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:27:40.122754 6839 raft_consensus.cc:504] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.123705 6839 leader_election.cc:288] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 7 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:40.123808 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2e443bad344ce89ce69871406c5a29" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.123978 6580 raft_consensus.cc:3037] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.124665 6840 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57e37f7e15b74819b692792eaed94f24"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:40.124830 6840 raft_consensus.cc:482] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.124944 6840 raft_consensus.cc:3037] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:40.126363 6580 raft_consensus.cc:2454] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 7.
I20240502 21:27:40.127074 6774 leader_election.cc:302] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:40.127329 9077 raft_consensus.cc:2781] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Leader election won for term 7
I20240502 21:27:40.127455 9077 raft_consensus.cc:686] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 7 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:40.127707 9077 consensus_queue.cc:237] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.128885 6577 raft_consensus.cc:2454] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 6.
I20240502 21:27:40.126907 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2e443bad344ce89ce69871406c5a29" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.129050 6968 raft_consensus.cc:729] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:40.129163 6970 raft_consensus.cc:3032] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 LEADER]: Stepping down as leader of term 6
I20240502 21:27:40.129253 6970 raft_consensus.cc:729] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:40.129503 6970 consensus_queue.cc:260] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.129729 6840 raft_consensus.cc:504] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.129850 6970 raft_consensus.cc:3037] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.130285 6439 catalog_manager.cc:5484] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 6 to 7, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 7 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.130558 6840 leader_election.cc:288] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 6 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:40.130775 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e37f7e15b74819b692792eaed94f24" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.130923 6708 raft_consensus.cc:3032] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:27:40.131000 6708 raft_consensus.cc:729] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:40.131264 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fbec2b1b80644a93b9239600157ad957"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:40.131232 6708 consensus_queue.cc:260] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.131421 6839 raft_consensus.cc:482] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.131507 6708 raft_consensus.cc:3037] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:40.131529 6839 raft_consensus.cc:3037] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:40.131902 6968 consensus_queue.cc:260] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.132246 6968 raft_consensus.cc:3037] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:40.132295 6970 raft_consensus.cc:2454] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 7.
I20240502 21:27:40.132642 6969 raft_consensus.cc:2454] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 7.
I20240502 21:27:40.133155 6774 leader_election.cc:302] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:40.133440 9077 raft_consensus.cc:2781] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:27:40.133420 6775 leader_election.cc:302] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters: ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:40.133607 9077 raft_consensus.cc:686] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 6 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:40.133864 9077 consensus_queue.cc:237] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.133987 6708 raft_consensus.cc:2396] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:27:40.134032 6839 raft_consensus.cc:504] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.134687 9077 raft_consensus.cc:2781] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Leader election won for term 7
I20240502 21:27:40.134814 9077 raft_consensus.cc:686] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 7 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:40.134851 6839 leader_election.cc:288] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 6 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:40.135028 9077 consensus_queue.cc:237] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.137887 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec2b1b80644a93b9239600157ad957" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.138078 6577 raft_consensus.cc:3037] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:40.138511 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e37f7e15b74819b692792eaed94f24" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.140565 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec2b1b80644a93b9239600157ad957" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.140926 6970 raft_consensus.cc:3032] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:27:40.141001 6970 raft_consensus.cc:729] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:40.141247 6970 consensus_queue.cc:260] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.141422 6840 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c6d11cba6ca04c17a9f47506824d9aa2"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:40.141527 6970 raft_consensus.cc:3037] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:40.141654 6840 raft_consensus.cc:482] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.141757 6840 raft_consensus.cc:3037] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.144142 6970 raft_consensus.cc:2454] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 6.
I20240502 21:27:40.144675 6775 leader_election.cc:302] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:40.144925 6840 raft_consensus.cc:504] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.145026 6969 raft_consensus.cc:3037] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:40.145426 9248 raft_consensus.cc:2781] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:27:40.146062 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d11cba6ca04c17a9f47506824d9aa2" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.146224 6708 raft_consensus.cc:3032] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 LEADER]: Stepping down as leader of term 6
I20240502 21:27:40.146337 6708 raft_consensus.cc:729] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:40.146533 6708 consensus_queue.cc:260] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.146790 6708 raft_consensus.cc:3037] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.147568 6969 raft_consensus.cc:2396] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:27:40.148403 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d11cba6ca04c17a9f47506824d9aa2" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.148641 6580 raft_consensus.cc:3037] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.149417 6708 raft_consensus.cc:2396] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240502 21:27:40.149359 6439 catalog_manager.cc:5484] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 5 to 6, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 6 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.145721 9248 raft_consensus.cc:686] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 6 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:40.148128 6775 leader_election.cc:302] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720; no voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:40.149878 6439 catalog_manager.cc:5484] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 6 to 7, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 7 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.150208 9074 raft_consensus.cc:2726] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240502 21:27:40.145823 6840 leader_election.cc:288] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 7 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:40.150539 9248 consensus_queue.cc:237] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.151021 6579 raft_consensus.cc:2454] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 7.
I20240502 21:27:40.151187 6840 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "444ccd95dc0f4bdd806ca9151d56d5cd"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:40.151348 6840 raft_consensus.cc:482] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.151464 6840 raft_consensus.cc:3037] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:40.153569 6840 raft_consensus.cc:504] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.154202 6840 leader_election.cc:288] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 6 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:40.154903 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444ccd95dc0f4bdd806ca9151d56d5cd" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.154970 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3093485f30b41939a1380ab7aa9e38b"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:40.155095 6708 raft_consensus.cc:3032] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:27:40.155186 6708 raft_consensus.cc:729] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:40.155176 6839 raft_consensus.cc:482] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.155321 6839 raft_consensus.cc:3037] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.153311 6580 raft_consensus.cc:2454] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 7.
I20240502 21:27:40.155460 6708 consensus_queue.cc:260] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.155740 6708 raft_consensus.cc:3037] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:40.156461 6577 raft_consensus.cc:2454] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 6.
I20240502 21:27:40.157086 6968 raft_consensus.cc:2454] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 6.
I20240502 21:27:40.157773 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444ccd95dc0f4bdd806ca9151d56d5cd" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.157930 6969 raft_consensus.cc:3037] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:40.158370 6708 raft_consensus.cc:2396] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:27:40.158800 6839 raft_consensus.cc:504] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.159586 6839 leader_election.cc:288] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 7 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:40.160506 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3093485f30b41939a1380ab7aa9e38b" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.160569 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3093485f30b41939a1380ab7aa9e38b" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.160717 6711 raft_consensus.cc:3032] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 LEADER]: Stepping down as leader of term 6
I20240502 21:27:40.160812 6711 raft_consensus.cc:729] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:40.161056 6774 leader_election.cc:302] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters: ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:40.161034 6711 consensus_queue.cc:260] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.161271 6968 raft_consensus.cc:3037] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.161314 6711 raft_consensus.cc:3037] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.162757 6439 catalog_manager.cc:5484] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 5 to 6, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 6 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:40.163697 6711 raft_consensus.cc:2396] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240502 21:27:40.164388 9248 raft_consensus.cc:2781] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Leader election won for term 7
I20240502 21:27:40.164655 9248 raft_consensus.cc:686] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 7 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:40.164881 6969 raft_consensus.cc:2396] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:27:40.164888 9248 consensus_queue.cc:237] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.165261 6775 leader_election.cc:302] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720; no voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:40.165637 9074 raft_consensus.cc:2726] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240502 21:27:40.165964 6968 raft_consensus.cc:2454] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 7.
I20240502 21:27:40.166333 6775 leader_election.cc:302] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters: ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:40.166581 9248 raft_consensus.cc:2781] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Leader election won for term 7
I20240502 21:27:40.166921 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1c98cbfbd0a44335b75b228c217f9132"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:40.167518 9248 raft_consensus.cc:686] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 7 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:40.167781 9248 consensus_queue.cc:237] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.168807 6439 catalog_manager.cc:5484] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 6 to 7, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 7 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.173843 6439 catalog_manager.cc:5484] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 6 to 7, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 7 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:40.174379 6839 raft_consensus.cc:482] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.174494 6839 raft_consensus.cc:3037] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:40.177181 6839 raft_consensus.cc:504] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.177740 6839 leader_election.cc:288] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 5 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:40.178118 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c98cbfbd0a44335b75b228c217f9132" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.178323 6577 raft_consensus.cc:3037] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:40.180717 6577 raft_consensus.cc:2454] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 5.
I20240502 21:27:40.180678 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c98cbfbd0a44335b75b228c217f9132" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.180872 6708 raft_consensus.cc:3032] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 LEADER]: Stepping down as leader of term 4
I20240502 21:27:40.180837 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ea3b52c673ae4b9f931059c4b0a456fa"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:40.180953 6708 raft_consensus.cc:729] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:40.181006 6839 raft_consensus.cc:482] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.181102 6774 leader_election.cc:302] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:40.181367 9248 raft_consensus.cc:2781] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:27:40.181234 6708 consensus_queue.cc:260] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.181115 6839 raft_consensus.cc:3037] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.181579 6708 raft_consensus.cc:3037] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:27:40.181732 9248 raft_consensus.cc:686] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [term 5 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:40.181979 9248 consensus_queue.cc:237] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.184023 6708 raft_consensus.cc:2396] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240502 21:27:40.183962 6839 raft_consensus.cc:504] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.184569 6839 leader_election.cc:288] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 7 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:40.184866 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3b52c673ae4b9f931059c4b0a456fa" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.185055 6969 raft_consensus.cc:3037] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.184971 6439 catalog_manager.cc:5484] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 4 to 5, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 5 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.185326 6840 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "169899da82594d9cb673f8db74764ca2"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:40.185493 6840 raft_consensus.cc:482] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.185580 6840 raft_consensus.cc:3037] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.187780 6969 raft_consensus.cc:2396] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:40.187983 6840 raft_consensus.cc:504] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.188508 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3b52c673ae4b9f931059c4b0a456fa" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.188706 6840 leader_election.cc:288] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 7 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:40.188736 6708 raft_consensus.cc:3032] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 LEADER]: Stepping down as leader of term 6
I20240502 21:27:40.188987 6708 raft_consensus.cc:729] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:40.189301 6708 consensus_queue.cc:260] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.189666 6708 raft_consensus.cc:3037] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.189664 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169899da82594d9cb673f8db74764ca2" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.189812 6577 raft_consensus.cc:3037] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.190531 6840 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8fac59b25c164e658fe71cfdf7ca78cb"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:40.190685 6840 raft_consensus.cc:482] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.190794 6840 raft_consensus.cc:3037] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:40.192103 6577 raft_consensus.cc:2454] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 7.
I20240502 21:27:40.192246 6708 raft_consensus.cc:2396] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:40.189375 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169899da82594d9cb673f8db74764ca2" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.192634 6711 raft_consensus.cc:3032] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 LEADER]: Stepping down as leader of term 6
I20240502 21:27:40.192736 6711 raft_consensus.cc:729] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:40.192971 6711 consensus_queue.cc:260] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.193204 6711 raft_consensus.cc:3037] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.193122 6840 raft_consensus.cc:504] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.193585 6840 leader_election.cc:288] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 6 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:40.194397 6840 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "093f8fd73ec34d8b9954c1716e910a78"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:40.194545 6840 raft_consensus.cc:482] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.194643 6840 raft_consensus.cc:3037] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:40.195479 6711 raft_consensus.cc:2396] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240502 21:27:40.195968 6776 leader_election.cc:302] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720; no voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:40.196157 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fac59b25c164e658fe71cfdf7ca78cb" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.196291 6711 raft_consensus.cc:3032] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:27:40.196378 6711 raft_consensus.cc:729] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:40.196478 6840 raft_consensus.cc:504] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.196611 9248 raft_consensus.cc:2726] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240502 21:27:40.196851 6711 consensus_queue.cc:260] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.196959 6840 leader_election.cc:288] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 6 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:40.197158 6711 raft_consensus.cc:3037] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:40.197580 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093f8fd73ec34d8b9954c1716e910a78" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.197818 6969 raft_consensus.cc:3037] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:40.199717 6711 raft_consensus.cc:2396] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:27:40.200280 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fac59b25c164e658fe71cfdf7ca78cb" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.200505 6577 raft_consensus.cc:3037] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:40.201692 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093f8fd73ec34d8b9954c1716e910a78" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.201838 6579 raft_consensus.cc:3037] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:40.202452 6969 raft_consensus.cc:2454] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 6.
I20240502 21:27:40.202800 6775 leader_election.cc:302] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:40.204257 6579 raft_consensus.cc:2396] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240502 21:27:40.205087 9248 raft_consensus.cc:2781] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:27:40.205257 9248 raft_consensus.cc:686] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 6 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:40.205507 9248 consensus_queue.cc:237] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.206383 6774 leader_election.cc:302] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters: ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:40.207083 9074 raft_consensus.cc:2781] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Leader election won for term 7
I20240502 21:27:40.207221 9074 raft_consensus.cc:686] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 7 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:40.207473 9074 consensus_queue.cc:237] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.210564 6577 raft_consensus.cc:2454] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 6.
I20240502 21:27:40.210981 6774 leader_election.cc:302] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters: ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:40.211270 9077 raft_consensus.cc:2781] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Leader election won for term 6
I20240502 21:27:40.211422 9077 raft_consensus.cc:686] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [term 6 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:40.211614 9077 consensus_queue.cc:237] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.220196 6439 catalog_manager.cc:5484] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 5 to 6, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 6 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.220786 6439 catalog_manager.cc:5484] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 6 to 7, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 7 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:40.221238 6439 catalog_manager.cc:5484] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 4 to 6, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 6 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:40.241159 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "df917e7196ed4939b75dc52930c87640"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:40.241348 6577 raft_consensus.cc:482] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.241480 6577 raft_consensus.cc:3037] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.243343 6577 raft_consensus.cc:504] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.244331 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df917e7196ed4939b75dc52930c87640" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.244340 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df917e7196ed4939b75dc52930c87640" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.244539 6711 raft_consensus.cc:3037] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.244541 6839 raft_consensus.cc:3032] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 6 LEADER]: Stepping down as leader of term 6
I20240502 21:27:40.244630 6839 raft_consensus.cc:729] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:40.244843 6839 consensus_queue.cc:260] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.245116 6839 raft_consensus.cc:3037] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.247144 6711 raft_consensus.cc:2396] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240502 21:27:40.247623 6577 leader_election.cc:288] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 7 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:40.248183 6839 raft_consensus.cc:2396] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240502 21:27:40.248633 6513 leader_election.cc:302] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae; no voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:40.248884 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:40.249068 8965 raft_consensus.cc:2726] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240502 21:27:40.249068 6577 raft_consensus.cc:482] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.249243 6577 raft_consensus.cc:3037] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.250743 6577 raft_consensus.cc:504] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.251188 6577 leader_election.cc:288] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 8 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:40.251780 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.251922 6839 raft_consensus.cc:3037] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.253589 6839 raft_consensus.cc:2454] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 8.
I20240502 21:27:40.251583 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.253996 6711 raft_consensus.cc:3037] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.253965 6513 leader_election.cc:302] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:40.254261 8965 raft_consensus.cc:2781] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Leader election won for term 8
I20240502 21:27:40.254412 8965 raft_consensus.cc:686] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:40.254686 8965 consensus_queue.cc:237] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.256183 6711 raft_consensus.cc:2396] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240502 21:27:40.256476 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3619e8a307de4583a4e470ad583fd7bc"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:40.256623 6577 raft_consensus.cc:482] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.256734 6577 raft_consensus.cc:3037] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.257952 6439 catalog_manager.cc:5484] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 6 to 8, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 8 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.259543 6577 raft_consensus.cc:504] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.260541 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3619e8a307de4583a4e470ad583fd7bc" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.260751 6969 raft_consensus.cc:3037] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.262390 6969 raft_consensus.cc:2454] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 8.
I20240502 21:27:40.262784 6577 leader_election.cc:288] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 8 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:40.262782 6514 leader_election.cc:302] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2; no voters:
I20240502 21:27:40.263152 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3619e8a307de4583a4e470ad583fd7bc" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.263300 6839 raft_consensus.cc:3032] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 7 LEADER]: Stepping down as leader of term 7
I20240502 21:27:40.263367 6839 raft_consensus.cc:729] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:40.263572 8965 raft_consensus.cc:2781] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Leader election won for term 8
I20240502 21:27:40.263609 6839 consensus_queue.cc:260] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.263724 8965 raft_consensus.cc:686] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:40.263855 6839 raft_consensus.cc:3037] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.263913 8965 consensus_queue.cc:237] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.266078 6839 raft_consensus.cc:2396] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240502 21:27:40.266351 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aae0bc634aa7424e935c8bc81a85099b"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:40.266546 6577 raft_consensus.cc:482] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.266687 6577 raft_consensus.cc:3037] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.267556 6439 catalog_manager.cc:5484] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 7 to 8, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 8 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:40.269816 6577 raft_consensus.cc:504] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.270809 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae0bc634aa7424e935c8bc81a85099b" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.271031 6711 raft_consensus.cc:3037] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.272646 6577 leader_election.cc:288] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 8 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:40.273034 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae0bc634aa7424e935c8bc81a85099b" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.273204 6969 raft_consensus.cc:3037] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.273473 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3b6b526ec34c41aea64a2a19f0208f5d"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:40.273604 6577 raft_consensus.cc:482] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.273689 6577 raft_consensus.cc:3037] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.275947 6577 raft_consensus.cc:504] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.276747 6969 raft_consensus.cc:2396] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240502 21:27:40.276952 6577 leader_election.cc:288] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 7 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:40.276970 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6b526ec34c41aea64a2a19f0208f5d" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.277141 6839 raft_consensus.cc:3032] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 6 LEADER]: Stepping down as leader of term 6
I20240502 21:27:40.277220 6839 raft_consensus.cc:729] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:40.277467 6839 consensus_queue.cc:260] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.277863 6839 raft_consensus.cc:3037] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.277915 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "da14dcc58d684704b3649e85b26fb43c"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:40.278070 6577 raft_consensus.cc:482] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.278157 6577 raft_consensus.cc:3037] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.280364 6577 raft_consensus.cc:504] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.281323 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da14dcc58d684704b3649e85b26fb43c" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.281329 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da14dcc58d684704b3649e85b26fb43c" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.277390 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6b526ec34c41aea64a2a19f0208f5d" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.281491 6840 raft_consensus.cc:3037] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.281491 6708 raft_consensus.cc:3037] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.281618 6968 raft_consensus.cc:3037] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.283850 6840 raft_consensus.cc:2454] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 8.
I20240502 21:27:40.283943 6708 raft_consensus.cc:2396] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240502 21:27:40.284302 6513 leader_election.cc:302] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:40.284582 6577 leader_election.cc:288] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 8 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:40.284595 9124 raft_consensus.cc:2781] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Leader election won for term 8
I20240502 21:27:40.284785 9124 raft_consensus.cc:686] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:40.285022 9124 consensus_queue.cc:237] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.285459 6968 raft_consensus.cc:2454] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 7.
I20240502 21:27:40.285795 6514 leader_election.cc:302] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2; no voters:
I20240502 21:27:40.286017 8965 raft_consensus.cc:2781] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Leader election won for term 7
I20240502 21:27:40.286155 8965 raft_consensus.cc:686] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 7 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:40.286367 8965 consensus_queue.cc:237] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.287688 6839 raft_consensus.cc:2396] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240502 21:27:40.288050 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "66198d766e6e4ceb820bc92bcca67d85"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:40.288220 6577 raft_consensus.cc:482] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.288326 6577 raft_consensus.cc:3037] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.288563 6711 raft_consensus.cc:2454] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 8.
I20240502 21:27:40.289127 6515 leader_election.cc:302] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:40.289412 9113 raft_consensus.cc:2781] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Leader election won for term 8
I20240502 21:27:40.289538 9113 raft_consensus.cc:686] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:40.289820 9113 consensus_queue.cc:237] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.291036 6577 raft_consensus.cc:504] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.291733 6577 leader_election.cc:288] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 8 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:40.292483 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66198d766e6e4ceb820bc92bcca67d85" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.292704 6969 raft_consensus.cc:3037] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.292878 6439 catalog_manager.cc:5484] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 6 to 8, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 8 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:40.293323 6439 catalog_manager.cc:5484] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 6 to 7, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 7 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.295603 6969 raft_consensus.cc:2454] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 8.
I20240502 21:27:40.295974 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66198d766e6e4ceb820bc92bcca67d85" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.296166 6839 raft_consensus.cc:3032] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 7 LEADER]: Stepping down as leader of term 7
I20240502 21:27:40.296242 6839 raft_consensus.cc:729] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:40.296494 6839 consensus_queue.cc:260] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.296739 6839 raft_consensus.cc:3037] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.296811 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f0b941baaab542a78748d4d0894cfbf4"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:40.296968 6577 raft_consensus.cc:482] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.297047 6577 raft_consensus.cc:3037] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.299681 6839 raft_consensus.cc:2396] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240502 21:27:40.299676 6439 catalog_manager.cc:5484] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 6 to 8, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 8 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.300143 6514 leader_election.cc:302] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2; no voters:
I20240502 21:27:40.300442 9113 raft_consensus.cc:2781] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Leader election won for term 8
I20240502 21:27:40.300566 9113 raft_consensus.cc:686] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:40.300812 9113 consensus_queue.cc:237] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.303959 6439 catalog_manager.cc:5484] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 7 to 8, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 8 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:40.304327 6577 raft_consensus.cc:504] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.305783 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b941baaab542a78748d4d0894cfbf4" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.305940 6839 raft_consensus.cc:3032] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 7 LEADER]: Stepping down as leader of term 7
I20240502 21:27:40.306000 6839 raft_consensus.cc:729] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:40.306206 6839 consensus_queue.cc:260] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.306406 6839 raft_consensus.cc:3037] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.310123 6839 raft_consensus.cc:2396] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20240502 21:27:40.310606 6577 leader_election.cc:288] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 8 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:40.311028 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b941baaab542a78748d4d0894cfbf4" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.311223 6711 raft_consensus.cc:3037] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.311432 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5e38d544ee14490c9fb5d8069440fd6a"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:40.311568 6577 raft_consensus.cc:482] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.311664 6577 raft_consensus.cc:3037] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.313468 6711 raft_consensus.cc:2454] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 8.
I20240502 21:27:40.314041 6515 leader_election.cc:302] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 9793130a8e4f45388357b8d2e9f47720
I20240502 21:27:40.314124 6577 raft_consensus.cc:504] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.314992 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e38d544ee14490c9fb5d8069440fd6a" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.315171 6839 raft_consensus.cc:3032] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 7 LEADER]: Stepping down as leader of term 7
I20240502 21:27:40.315202 9124 raft_consensus.cc:2781] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Leader election won for term 8
I20240502 21:27:40.315236 6839 raft_consensus.cc:729] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:40.315361 9124 raft_consensus.cc:686] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:40.315508 6839 consensus_queue.cc:260] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.315757 6839 raft_consensus.cc:3037] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.315696 9124 consensus_queue.cc:237] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.318260 6839 raft_consensus.cc:2396] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240502 21:27:40.319744 6439 catalog_manager.cc:5484] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 7 to 8, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 8 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.322018 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e38d544ee14490c9fb5d8069440fd6a" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.322216 6969 raft_consensus.cc:3037] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.324743 6969 raft_consensus.cc:2454] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 8.
I20240502 21:27:40.325160 6514 leader_election.cc:302] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2; no voters: 9793130a8e4f45388357b8d2e9f47720
I20240502 21:27:40.325424 9113 raft_consensus.cc:2781] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Leader election won for term 8
I20240502 21:27:40.325565 9113 raft_consensus.cc:686] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:40.325855 6577 leader_election.cc:288] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 8 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:40.325790 9113 consensus_queue.cc:237] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.327358 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5b2e443bad344ce89ce69871406c5a29"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:40.327546 6577 raft_consensus.cc:482] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.327666 6577 raft_consensus.cc:3037] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.328153 6439 catalog_manager.cc:5484] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 7 to 8, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 8 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.331492 6577 raft_consensus.cc:504] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.332418 6577 leader_election.cc:288] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 8 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:40.332654 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2e443bad344ce89ce69871406c5a29" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.332691 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2e443bad344ce89ce69871406c5a29" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.332830 6839 raft_consensus.cc:3032] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 7 LEADER]: Stepping down as leader of term 7
I20240502 21:27:40.332859 6969 raft_consensus.cc:3037] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.332916 6839 raft_consensus.cc:729] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:40.333192 6839 consensus_queue.cc:260] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.333452 6839 raft_consensus.cc:3037] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.334497 6969 raft_consensus.cc:2454] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 8.
I20240502 21:27:40.334892 6514 leader_election.cc:302] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2; no voters:
I20240502 21:27:40.335320 9113 raft_consensus.cc:2781] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Leader election won for term 8
I20240502 21:27:40.335494 9113 raft_consensus.cc:686] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:40.335534 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fb023d72aa8d4248a64d4723e18a3cf6"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:40.335671 6577 raft_consensus.cc:482] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.335806 6577 raft_consensus.cc:3037] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.335747 9113 consensus_queue.cc:237] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.335997 6839 raft_consensus.cc:2396] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240502 21:27:40.338233 6577 raft_consensus.cc:504] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.339583 6577 leader_election.cc:288] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 8 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:40.339658 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb023d72aa8d4248a64d4723e18a3cf6" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.339829 6969 raft_consensus.cc:3037] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.339865 6439 catalog_manager.cc:5484] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 7 to 8, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 8 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.340421 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c6d11cba6ca04c17a9f47506824d9aa2"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:40.340601 6577 raft_consensus.cc:482] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.340682 6577 raft_consensus.cc:3037] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.340807 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb023d72aa8d4248a64d4723e18a3cf6" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.340998 6711 raft_consensus.cc:3037] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.342324 6969 raft_consensus.cc:2396] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240502 21:27:40.343047 6577 raft_consensus.cc:504] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.343425 6711 raft_consensus.cc:2454] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 8.
I20240502 21:27:40.343859 6577 leader_election.cc:288] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 8 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:40.344823 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "331bcde3e6bf43c081af836c8cdebdc7"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:40.345001 6577 raft_consensus.cc:482] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.345126 6577 raft_consensus.cc:3037] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.345669 6515 leader_election.cc:302] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:40.346253 9113 raft_consensus.cc:2781] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Leader election won for term 8
I20240502 21:27:40.346367 9113 raft_consensus.cc:686] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:40.346587 9113 consensus_queue.cc:237] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.348269 6577 raft_consensus.cc:504] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.349267 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331bcde3e6bf43c081af836c8cdebdc7" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.349514 6969 raft_consensus.cc:3037] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.349992 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d11cba6ca04c17a9f47506824d9aa2" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.350193 6839 raft_consensus.cc:3032] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 7 LEADER]: Stepping down as leader of term 7
I20240502 21:27:40.350270 6839 raft_consensus.cc:729] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:40.350495 6839 consensus_queue.cc:260] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.350744 6839 raft_consensus.cc:3037] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.352706 6969 raft_consensus.cc:2396] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240502 21:27:40.353487 6577 leader_election.cc:288] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 7 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:40.353977 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331bcde3e6bf43c081af836c8cdebdc7" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.354185 6708 raft_consensus.cc:3037] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.354233 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d11cba6ca04c17a9f47506824d9aa2" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.354431 6711 raft_consensus.cc:3037] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.356629 6708 raft_consensus.cc:2454] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 7.
I20240502 21:27:40.357182 6515 leader_election.cc:302] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:40.357240 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fbec2b1b80644a93b9239600157ad957"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:40.357460 6577 raft_consensus.cc:482] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.357538 9124 raft_consensus.cc:2781] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Leader election won for term 7
I20240502 21:27:40.357589 6577 raft_consensus.cc:3037] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.357710 9124 raft_consensus.cc:686] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 7 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:40.357949 9124 consensus_queue.cc:237] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.359411 6839 raft_consensus.cc:2396] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240502 21:27:40.359943 6577 raft_consensus.cc:504] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.360635 6577 leader_election.cc:288] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 7 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:40.360852 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec2b1b80644a93b9239600157ad957" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.360901 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec2b1b80644a93b9239600157ad957" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.361053 6969 raft_consensus.cc:3037] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.361083 6839 raft_consensus.cc:3032] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 6 LEADER]: Stepping down as leader of term 6
I20240502 21:27:40.361189 6839 raft_consensus.cc:729] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:40.361514 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1c98cbfbd0a44335b75b228c217f9132"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:40.361477 6839 consensus_queue.cc:260] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.361647 6577 raft_consensus.cc:482] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.361703 6839 raft_consensus.cc:3037] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.361747 6577 raft_consensus.cc:3037] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:40.363529 6969 raft_consensus.cc:2454] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 7.
I20240502 21:27:40.363958 6514 leader_election.cc:302] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2; no voters:
I20240502 21:27:40.364063 6577 raft_consensus.cc:504] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.364862 6577 leader_election.cc:288] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 6 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:40.364929 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c98cbfbd0a44335b75b228c217f9132" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.365154 6840 raft_consensus.cc:3032] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 21:27:40.365223 6840 raft_consensus.cc:729] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:40.365553 6840 consensus_queue.cc:260] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.365732 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "169899da82594d9cb673f8db74764ca2"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:40.365864 6840 raft_consensus.cc:3037] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:40.365888 6577 raft_consensus.cc:482] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.365974 6577 raft_consensus.cc:3037] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.368343 6577 raft_consensus.cc:504] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.368993 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c98cbfbd0a44335b75b228c217f9132" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.369171 6708 raft_consensus.cc:3037] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 5 FOLLOWER]: Advancing to term 6
I20240502 21:27:40.369874 6840 raft_consensus.cc:2396] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240502 21:27:40.371562 6708 raft_consensus.cc:2396] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240502 21:27:40.372097 6515 leader_election.cc:302] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae; no voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:40.372403 6577 leader_election.cc:288] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 8 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:40.372686 9124 raft_consensus.cc:2781] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Leader election won for term 7
I20240502 21:27:40.372730 8965 raft_consensus.cc:2726] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240502 21:27:40.373251 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169899da82594d9cb673f8db74764ca2" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.373438 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169899da82594d9cb673f8db74764ca2" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.373493 6840 raft_consensus.cc:3032] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 7 LEADER]: Stepping down as leader of term 7
I20240502 21:27:40.373574 6840 raft_consensus.cc:729] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:40.373633 6708 raft_consensus.cc:3037] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.373873 6840 consensus_queue.cc:260] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.374193 6840 raft_consensus.cc:3037] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.375340 6711 raft_consensus.cc:2396] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240502 21:27:40.375753 6515 leader_election.cc:302] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae; no voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:40.376001 6708 raft_consensus.cc:2396] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240502 21:27:40.376343 8965 raft_consensus.cc:2726] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240502 21:27:40.376390 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8fac59b25c164e658fe71cfdf7ca78cb"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:40.376554 6577 raft_consensus.cc:482] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.376664 6577 raft_consensus.cc:3037] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.379537 6839 raft_consensus.cc:2396] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240502 21:27:40.380095 6577 raft_consensus.cc:504] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.380946 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fac59b25c164e658fe71cfdf7ca78cb" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.381150 6841 raft_consensus.cc:3032] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [term 6 LEADER]: Stepping down as leader of term 6
I20240502 21:27:40.381232 6841 raft_consensus.cc:729] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:40.381480 6841 consensus_queue.cc:260] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.381757 6841 raft_consensus.cc:3037] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.382812 6840 raft_consensus.cc:2396] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240502 21:27:40.384300 6841 raft_consensus.cc:2396] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:40.384721 6513 leader_election.cc:302] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae; no voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:40.385109 8965 raft_consensus.cc:2726] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240502 21:27:40.386334 6577 leader_election.cc:288] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 7 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:40.386642 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fac59b25c164e658fe71cfdf7ca78cb" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.386813 6711 raft_consensus.cc:3037] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.387250 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "093f8fd73ec34d8b9954c1716e910a78"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:40.387522 6577 raft_consensus.cc:482] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.387681 6577 raft_consensus.cc:3037] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.389066 9124 raft_consensus.cc:686] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 7 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:40.389381 9124 consensus_queue.cc:237] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.390233 6577 raft_consensus.cc:504] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.391129 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093f8fd73ec34d8b9954c1716e910a78" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.391264 6711 raft_consensus.cc:2396] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:40.391348 6969 raft_consensus.cc:3037] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.391742 6515 leader_election.cc:302] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae; no voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:40.392325 9124 raft_consensus.cc:2726] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240502 21:27:40.393867 6969 raft_consensus.cc:2454] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 7.
I20240502 21:27:40.394277 6514 leader_election.cc:302] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2; no voters:
I20240502 21:27:40.394537 9113 raft_consensus.cc:2781] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Leader election won for term 7
I20240502 21:27:40.394774 9113 raft_consensus.cc:686] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 7 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:40.394781 6577 leader_election.cc:288] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 7 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:40.395107 9113 consensus_queue.cc:237] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.395406 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093f8fd73ec34d8b9954c1716e910a78" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.395583 6841 raft_consensus.cc:3032] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 6 LEADER]: Stepping down as leader of term 6
I20240502 21:27:40.395651 6841 raft_consensus.cc:729] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:40.395869 6841 consensus_queue.cc:260] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.396106 6841 raft_consensus.cc:3037] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.398490 6841 raft_consensus.cc:2396] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240502 21:27:40.399214 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1325bd7bbb4340f88bdc7c705934862e"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:40.399389 6577 raft_consensus.cc:482] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.399513 6577 raft_consensus.cc:3037] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.402278 6577 raft_consensus.cc:504] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.403273 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1325bd7bbb4340f88bdc7c705934862e" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.403286 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1325bd7bbb4340f88bdc7c705934862e" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.403457 6969 raft_consensus.cc:3037] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.403528 6711 raft_consensus.cc:3037] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.398499 6439 catalog_manager.cc:5484] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 6 to 8, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 8 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.405908 6969 raft_consensus.cc:2396] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240502 21:27:40.406373 6577 leader_election.cc:288] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 8 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:40.406774 6711 raft_consensus.cc:2454] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 8.
I20240502 21:27:40.407181 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "875f83053fee45b3bc1921b24745e9da"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:40.407368 6577 raft_consensus.cc:482] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.407320 6515 leader_election.cc:302] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:40.407500 6577 raft_consensus.cc:3037] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.407863 9124 raft_consensus.cc:2781] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Leader election won for term 8
I20240502 21:27:40.408051 9124 raft_consensus.cc:686] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:40.408421 9124 consensus_queue.cc:237] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.410135 6577 raft_consensus.cc:504] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.411021 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875f83053fee45b3bc1921b24745e9da" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.411204 6969 raft_consensus.cc:3037] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.413508 6969 raft_consensus.cc:2454] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 7.
I20240502 21:27:40.413859 6514 leader_election.cc:302] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2; no voters:
I20240502 21:27:40.414129 9273 raft_consensus.cc:2781] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Leader election won for term 7
I20240502 21:27:40.414258 9273 raft_consensus.cc:686] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 7 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:40.414510 9273 consensus_queue.cc:237] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.415927 6577 leader_election.cc:288] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 7 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:40.416301 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875f83053fee45b3bc1921b24745e9da" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.416522 6711 raft_consensus.cc:3032] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 LEADER]: Stepping down as leader of term 6
I20240502 21:27:40.416641 6711 raft_consensus.cc:729] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:40.416955 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "533d0913d0bc4582b976a58bda8176bc"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:40.416971 6711 consensus_queue.cc:260] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.417156 6577 raft_consensus.cc:482] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.417369 6711 raft_consensus.cc:3037] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.417390 6577 raft_consensus.cc:3037] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.419941 6577 raft_consensus.cc:504] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.420931 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533d0913d0bc4582b976a58bda8176bc" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.421129 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533d0913d0bc4582b976a58bda8176bc" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.421188 6708 raft_consensus.cc:3037] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.421298 6969 raft_consensus.cc:3037] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.423645 6969 raft_consensus.cc:2396] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240502 21:27:40.424223 6577 leader_election.cc:288] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 8 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:40.424463 6708 raft_consensus.cc:2454] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 8.
I20240502 21:27:40.424927 6515 leader_election.cc:302] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:40.425180 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c9e728880af44303a975a63a4bd7bc62"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:40.425442 6577 raft_consensus.cc:482] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.425473 8965 raft_consensus.cc:2781] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Leader election won for term 8
I20240502 21:27:40.425648 6577 raft_consensus.cc:3037] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.425695 8965 raft_consensus.cc:686] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:40.426587 8965 consensus_queue.cc:237] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.428301 6577 raft_consensus.cc:504] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.429306 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e728880af44303a975a63a4bd7bc62" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.429548 6708 raft_consensus.cc:3032] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 LEADER]: Stepping down as leader of term 7
I20240502 21:27:40.429666 6708 raft_consensus.cc:729] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:40.429958 6708 consensus_queue.cc:260] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.430403 6708 raft_consensus.cc:3037] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.432807 6708 raft_consensus.cc:2396] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240502 21:27:40.433447 6577 leader_election.cc:288] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 8 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:40.433918 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e728880af44303a975a63a4bd7bc62" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.434114 6969 raft_consensus.cc:3037] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.436409 6969 raft_consensus.cc:2454] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 8.
I20240502 21:27:40.437460 6514 leader_election.cc:302] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2; no voters: ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:40.437813 9273 raft_consensus.cc:2781] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Leader election won for term 8
I20240502 21:27:40.442438 6711 raft_consensus.cc:2396] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240502 21:27:40.443583 6969 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3093485f30b41939a1380ab7aa9e38b"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:40.443804 6969 raft_consensus.cc:482] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.443949 6969 raft_consensus.cc:3037] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.447070 6969 raft_consensus.cc:504] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.448163 6969 leader_election.cc:288] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 8 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:40.448259 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3093485f30b41939a1380ab7aa9e38b" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.448551 6708 raft_consensus.cc:3037] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.448801 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3093485f30b41939a1380ab7aa9e38b" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.449029 6841 raft_consensus.cc:3032] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 7 LEADER]: Stepping down as leader of term 7
I20240502 21:27:40.449108 6841 raft_consensus.cc:729] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:40.449188 6969 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b2b3e30b94d47dd822539cb30f36f81"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:40.449348 6969 raft_consensus.cc:482] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.449431 6969 raft_consensus.cc:3037] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.449385 6841 consensus_queue.cc:260] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.449702 6841 raft_consensus.cc:3037] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.451349 6708 raft_consensus.cc:2396] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240502 21:27:40.452319 6841 raft_consensus.cc:2396] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240502 21:27:40.452292 6969 raft_consensus.cc:504] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.453001 6969 leader_election.cc:288] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 8 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:40.453538 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2b3e30b94d47dd822539cb30f36f81" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.453732 6708 raft_consensus.cc:3037] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.452996 6904 leader_election.cc:302] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2; no voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:40.454614 9106 raft_consensus.cc:2726] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240502 21:27:40.455555 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2b3e30b94d47dd822539cb30f36f81" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.455775 6841 raft_consensus.cc:3032] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 7 LEADER]: Stepping down as leader of term 7
I20240502 21:27:40.455871 6841 raft_consensus.cc:729] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:40.456188 6841 consensus_queue.cc:260] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.456562 6841 raft_consensus.cc:3037] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.456825 6708 raft_consensus.cc:2454] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 8.
I20240502 21:27:40.457175 6906 leader_election.cc:302] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:40.457430 9106 raft_consensus.cc:2781] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Leader election won for term 8
I20240502 21:27:40.457623 9106 raft_consensus.cc:686] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:40.457871 9106 consensus_queue.cc:237] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.460602 6841 raft_consensus.cc:2396] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240502 21:27:40.460703 6969 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aae0bc634aa7424e935c8bc81a85099b"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:40.460870 6969 raft_consensus.cc:482] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.461010 6969 raft_consensus.cc:3037] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.461107 6436 catalog_manager.cc:5484] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 7 to 8, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 8 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:40.462954 6969 raft_consensus.cc:504] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.463845 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae0bc634aa7424e935c8bc81a85099b" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.463884 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae0bc634aa7424e935c8bc81a85099b" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.464118 6577 raft_consensus.cc:3032] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Stepping down as leader of term 8
I20240502 21:27:40.464164 6708 raft_consensus.cc:3037] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.464267 6577 raft_consensus.cc:729] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:40.464574 6577 consensus_queue.cc:260] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.464865 6577 raft_consensus.cc:3037] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.467381 6708 raft_consensus.cc:2454] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 9.
I20240502 21:27:40.467841 6906 leader_election.cc:302] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:40.468084 9106 raft_consensus.cc:2781] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Leader election won for term 9
I20240502 21:27:40.468252 9106 raft_consensus.cc:686] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 9 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:40.468587 9106 consensus_queue.cc:237] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.473857 6969 leader_election.cc:288] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 9 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:40.474861 6968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3619e8a307de4583a4e470ad583fd7bc"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:40.475106 6968 raft_consensus.cc:482] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.475214 6968 raft_consensus.cc:3037] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.478370 6968 raft_consensus.cc:504] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.479856 6968 leader_election.cc:288] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 9 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:40.480888 6968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3b6b526ec34c41aea64a2a19f0208f5d"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:40.481117 6968 raft_consensus.cc:482] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.481274 6968 raft_consensus.cc:3037] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.484560 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3619e8a307de4583a4e470ad583fd7bc" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.486078 6841 raft_consensus.cc:3037] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.486665 6436 catalog_manager.cc:5484] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 8 to 9, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 9 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:40.489053 6577 raft_consensus.cc:2396] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240502 21:27:40.489152 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3619e8a307de4583a4e470ad583fd7bc" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.490365 6579 raft_consensus.cc:3032] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Stepping down as leader of term 8
I20240502 21:27:40.490465 6579 raft_consensus.cc:729] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:40.484274 6968 raft_consensus.cc:504] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.490717 6579 consensus_queue.cc:260] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.490909 6579 raft_consensus.cc:3037] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.491281 6968 leader_election.cc:288] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 8 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:40.491703 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6b526ec34c41aea64a2a19f0208f5d" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.491876 6839 raft_consensus.cc:3037] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.492249 6968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c10bca264dda4180bd19beb5767bf047"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:40.492494 6968 raft_consensus.cc:482] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.492460 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6b526ec34c41aea64a2a19f0208f5d" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.492615 6968 raft_consensus.cc:3037] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.492662 6577 raft_consensus.cc:3032] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 7 LEADER]: Stepping down as leader of term 7
I20240502 21:27:40.492749 6577 raft_consensus.cc:729] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:40.492996 6577 consensus_queue.cc:260] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.493211 6841 raft_consensus.cc:2396] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240502 21:27:40.493309 6577 raft_consensus.cc:3037] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.493552 6579 raft_consensus.cc:2396] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240502 21:27:40.495491 9273 raft_consensus.cc:686] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:40.495853 6577 raft_consensus.cc:2396] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240502 21:27:40.496385 6904 leader_election.cc:302] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2; no voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720
I20240502 21:27:40.496872 6968 raft_consensus.cc:504] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.498016 9106 raft_consensus.cc:2726] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240502 21:27:40.498026 6968 leader_election.cc:288] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 7 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:40.495796 9273 consensus_queue.cc:237] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.499910 6968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "66198d766e6e4ceb820bc92bcca67d85"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:40.500110 6968 raft_consensus.cc:482] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.500241 6968 raft_consensus.cc:3037] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.500844 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10bca264dda4180bd19beb5767bf047" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.501123 6708 raft_consensus.cc:2379] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 in current term 7: Already voted for candidate 9793130a8e4f45388357b8d2e9f47720 in this term.
I20240502 21:27:40.503134 6436 catalog_manager.cc:5484] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 6 to 8, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 8 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.503732 6968 raft_consensus.cc:504] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.503861 6436 catalog_manager.cc:5484] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 5 to 7, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 7 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.504503 6436 catalog_manager.cc:5484] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 6 to 7, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 7 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.505079 6436 catalog_manager.cc:5484] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 6 to 7, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 7 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.505149 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66198d766e6e4ceb820bc92bcca67d85" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.506402 6577 raft_consensus.cc:3032] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Stepping down as leader of term 8
I20240502 21:27:40.506531 6577 raft_consensus.cc:729] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:40.506791 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66198d766e6e4ceb820bc92bcca67d85" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.507040 6840 raft_consensus.cc:3037] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.507246 6577 consensus_queue.cc:260] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.507637 6577 raft_consensus.cc:3037] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.508643 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10bca264dda4180bd19beb5767bf047" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.508935 6841 raft_consensus.cc:2379] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 7 LEADER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 in current term 7: Already voted for candidate 9793130a8e4f45388357b8d2e9f47720 in this term.
I20240502 21:27:40.509466 6904 leader_election.cc:302] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2; no voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:40.509758 9106 raft_consensus.cc:2726] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240502 21:27:40.510977 6577 raft_consensus.cc:2396] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240502 21:27:40.511869 6840 raft_consensus.cc:2396] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240502 21:27:40.513042 6839 raft_consensus.cc:2396] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240502 21:27:40.513492 6904 leader_election.cc:302] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2; no voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720
I20240502 21:27:40.513798 9106 raft_consensus.cc:2726] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240502 21:27:40.515658 6968 leader_election.cc:288] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 9 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:40.517683 6968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "331bcde3e6bf43c081af836c8cdebdc7"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:40.517932 6968 raft_consensus.cc:482] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.518060 6968 raft_consensus.cc:3037] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.520649 6436 catalog_manager.cc:5484] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 7 to 8, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 8 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:40.521373 6436 catalog_manager.cc:5484] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 6 to 7, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 7 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.522063 6436 catalog_manager.cc:5484] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 6 to 8, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 8 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.522593 6968 raft_consensus.cc:504] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.523348 6968 leader_election.cc:288] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 8 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:40.523933 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331bcde3e6bf43c081af836c8cdebdc7" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.524173 6708 raft_consensus.cc:3037] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.524284 6969 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57e37f7e15b74819b692792eaed94f24"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:40.524524 6969 raft_consensus.cc:482] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.524634 6969 raft_consensus.cc:3037] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.527395 6708 raft_consensus.cc:2454] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 8.
I20240502 21:27:40.527590 6969 raft_consensus.cc:504] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.527789 6906 leader_election.cc:302] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:40.528064 9106 raft_consensus.cc:2781] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Leader election won for term 8
I20240502 21:27:40.528226 6969 leader_election.cc:288] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 7 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:40.528223 9106 raft_consensus.cc:686] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:40.529012 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e37f7e15b74819b692792eaed94f24" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.528998 9106 consensus_queue.cc:237] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.529210 6839 raft_consensus.cc:3037] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.530969 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e37f7e15b74819b692792eaed94f24" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.530956 6969 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fbec2b1b80644a93b9239600157ad957"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:40.531190 6708 raft_consensus.cc:3037] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.531193 6969 raft_consensus.cc:482] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.531334 6969 raft_consensus.cc:3037] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.531536 6839 raft_consensus.cc:2454] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 7.
I20240502 21:27:40.531996 6904 leader_election.cc:302] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:40.532289 9002 raft_consensus.cc:2781] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Leader election won for term 7
I20240502 21:27:40.532456 9002 raft_consensus.cc:686] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:40.532727 9002 consensus_queue.cc:237] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.534163 6708 raft_consensus.cc:2396] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240502 21:27:40.534301 6969 raft_consensus.cc:504] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.535396 6969 leader_election.cc:288] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 8 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:40.535494 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec2b1b80644a93b9239600157ad957" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.535686 6839 raft_consensus.cc:3037] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.537051 6969 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ea3b52c673ae4b9f931059c4b0a456fa"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:40.537349 6969 raft_consensus.cc:482] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.537461 6969 raft_consensus.cc:3037] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.537519 6436 catalog_manager.cc:5484] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 7 to 8, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 8 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.538512 6839 raft_consensus.cc:2396] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240502 21:27:40.542297 6969 raft_consensus.cc:504] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.543277 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3b52c673ae4b9f931059c4b0a456fa" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.543511 6708 raft_consensus.cc:3037] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.543733 6904 leader_election.cc:302] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2; no voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720
I20240502 21:27:40.544027 9002 raft_consensus.cc:2726] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240502 21:27:40.544777 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec2b1b80644a93b9239600157ad957" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.544811 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331bcde3e6bf43c081af836c8cdebdc7" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.544997 6579 raft_consensus.cc:3032] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 7 LEADER]: Stepping down as leader of term 7
I20240502 21:27:40.545058 6577 raft_consensus.cc:3032] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 7 LEADER]: Stepping down as leader of term 7
I20240502 21:27:40.545074 6579 raft_consensus.cc:729] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:40.545218 6577 raft_consensus.cc:729] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:40.545469 6579 consensus_queue.cc:260] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.545508 6577 consensus_queue.cc:260] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.545740 6579 raft_consensus.cc:3037] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.545843 6577 raft_consensus.cc:3037] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.546438 6708 raft_consensus.cc:2396] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240502 21:27:40.546950 6436 catalog_manager.cc:5484] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 5 to 7, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 7 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.550335 6577 raft_consensus.cc:2396] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240502 21:27:40.550591 6579 raft_consensus.cc:2396] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240502 21:27:40.551607 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3b52c673ae4b9f931059c4b0a456fa" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.551822 6839 raft_consensus.cc:3037] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.554549 6969 leader_election.cc:288] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 8 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:40.555451 6968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fb023d72aa8d4248a64d4723e18a3cf6"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:40.555634 6968 raft_consensus.cc:482] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.555756 6968 raft_consensus.cc:3037] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.557726 6839 raft_consensus.cc:2454] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 8.
I20240502 21:27:40.558209 6904 leader_election.cc:302] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters: ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:40.558537 9002 raft_consensus.cc:2781] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Leader election won for term 8
I20240502 21:27:40.558701 9002 raft_consensus.cc:686] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 8 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:40.558954 9002 consensus_queue.cc:237] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.559715 6968 raft_consensus.cc:504] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.559789 6904 leader_election.cc:302] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2; no voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720
I20240502 21:27:40.560616 9106 raft_consensus.cc:2726] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240502 21:27:40.560612 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb023d72aa8d4248a64d4723e18a3cf6" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.560686 6968 leader_election.cc:288] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 9 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:40.560840 6708 raft_consensus.cc:3037] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.561671 6968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "980f9cc0cdc6452d8009e9f75a0d4275"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:40.561950 6968 raft_consensus.cc:482] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.562127 6968 raft_consensus.cc:3037] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.562346 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb023d72aa8d4248a64d4723e18a3cf6" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.562526 6577 raft_consensus.cc:3032] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Stepping down as leader of term 8
I20240502 21:27:40.562618 6577 raft_consensus.cc:729] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:40.562868 6577 consensus_queue.cc:260] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.563148 6577 raft_consensus.cc:3037] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.563256 6708 raft_consensus.cc:2454] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 9.
I20240502 21:27:40.563630 6906 leader_election.cc:302] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:40.564021 9106 raft_consensus.cc:2781] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Leader election won for term 9
I20240502 21:27:40.564215 9106 raft_consensus.cc:686] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 9 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:40.564522 9106 consensus_queue.cc:237] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.571311 6968 raft_consensus.cc:504] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.572474 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "980f9cc0cdc6452d8009e9f75a0d4275" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.572755 6839 raft_consensus.cc:3037] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.572974 6577 raft_consensus.cc:2396] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240502 21:27:40.575619 6839 raft_consensus.cc:2454] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 8.
I20240502 21:27:40.576073 6904 leader_election.cc:302] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:40.576442 9002 raft_consensus.cc:2781] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Leader election won for term 8
I20240502 21:27:40.576162 6436 catalog_manager.cc:5484] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 6 to 8, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 8 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.576802 9002 raft_consensus.cc:686] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:40.577229 9002 consensus_queue.cc:237] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.584823 6968 leader_election.cc:288] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 8 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:40.584692 6436 catalog_manager.cc:5484] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 8 to 9, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 9 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:40.585222 6436 catalog_manager.cc:5484] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 6 to 8, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 8 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.586390 6968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5e38d544ee14490c9fb5d8069440fd6a"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:40.586550 6968 raft_consensus.cc:482] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.586653 6968 raft_consensus.cc:3037] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.589612 6969 raft_consensus.cc:1261] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Refusing update from remote peer 9793130a8e4f45388357b8d2e9f47720: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240502 21:27:40.590231 9077 consensus_queue.cc:1035] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:40.596879 6968 raft_consensus.cc:504] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.598527 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e38d544ee14490c9fb5d8069440fd6a" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.598754 6839 raft_consensus.cc:3037] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.599386 6558 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:40.602253 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "980f9cc0cdc6452d8009e9f75a0d4275" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.602502 6708 raft_consensus.cc:3037] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.602802 6839 raft_consensus.cc:2396] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240502 21:27:40.605295 6968 leader_election.cc:288] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 9 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:40.605410 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e38d544ee14490c9fb5d8069440fd6a" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.605609 6577 raft_consensus.cc:3032] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Stepping down as leader of term 8
I20240502 21:27:40.605711 6577 raft_consensus.cc:729] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:40.605986 6577 consensus_queue.cc:260] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.606150 6969 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a09d2491d9dc4cbfb20f7adc1087b375"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:40.606302 6577 raft_consensus.cc:3037] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.606338 6969 raft_consensus.cc:482] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.606424 6969 raft_consensus.cc:3037] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.607030 6708 raft_consensus.cc:2396] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240502 21:27:40.608668 6708 raft_consensus.cc:1261] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Refusing update from remote peer 9793130a8e4f45388357b8d2e9f47720: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240502 21:27:40.609112 6969 raft_consensus.cc:504] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.609973 9248 consensus_queue.cc:1035] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:40.610013 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09d2491d9dc4cbfb20f7adc1087b375" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.610260 6839 raft_consensus.cc:3032] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 7 LEADER]: Stepping down as leader of term 7
I20240502 21:27:40.610347 6839 raft_consensus.cc:729] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:40.610602 6839 consensus_queue.cc:260] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.610882 6839 raft_consensus.cc:3037] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.611137 6577 raft_consensus.cc:2396] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240502 21:27:40.611778 6904 leader_election.cc:302] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2; no voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720
I20240502 21:27:40.612840 9002 raft_consensus.cc:2726] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240502 21:27:40.613456 6839 raft_consensus.cc:2396] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240502 21:27:40.614167 6969 leader_election.cc:288] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 8 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:40.614971 6970 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5b2e443bad344ce89ce69871406c5a29"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:40.615168 6970 raft_consensus.cc:482] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.615269 6970 raft_consensus.cc:3037] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.617700 6970 raft_consensus.cc:504] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.618281 6970 leader_election.cc:288] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 9 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:40.619422 6970 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "093f8fd73ec34d8b9954c1716e910a78"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:40.619580 6970 raft_consensus.cc:482] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.619717 6970 raft_consensus.cc:3037] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.619768 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2e443bad344ce89ce69871406c5a29" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.619962 6840 raft_consensus.cc:3037] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.622357 6970 raft_consensus.cc:504] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.622929 6840 raft_consensus.cc:2396] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240502 21:27:40.623658 6970 leader_election.cc:288] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 8 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:40.623894 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093f8fd73ec34d8b9954c1716e910a78" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 8 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.624095 6579 raft_consensus.cc:3032] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 7 LEADER]: Stepping down as leader of term 7
I20240502 21:27:40.624177 6579 raft_consensus.cc:729] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:40.624480 6579 consensus_queue.cc:260] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.624641 6969 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1325bd7bbb4340f88bdc7c705934862e"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:40.624742 6579 raft_consensus.cc:3037] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.624817 6969 raft_consensus.cc:482] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.624943 6969 raft_consensus.cc:3037] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.626684 6579 raft_consensus.cc:2396] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:27:40.627362 6969 raft_consensus.cc:504] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.627583 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093f8fd73ec34d8b9954c1716e910a78" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 8 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.627781 6839 raft_consensus.cc:3037] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.628384 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1325bd7bbb4340f88bdc7c705934862e" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.628625 6711 raft_consensus.cc:3037] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.630151 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1325bd7bbb4340f88bdc7c705934862e" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.630383 6579 raft_consensus.cc:3032] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Stepping down as leader of term 8
I20240502 21:27:40.630467 6579 raft_consensus.cc:729] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:40.630782 6579 consensus_queue.cc:260] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.631283 6711 raft_consensus.cc:2454] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 9.
I20240502 21:27:40.631516 8750 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:40.632580 6906 leader_election.cc:302] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:40.632936 9002 raft_consensus.cc:2781] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Leader election won for term 9
I20240502 21:27:40.633183 9002 raft_consensus.cc:686] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 9 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:40.633436 9002 consensus_queue.cc:237] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.633765 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2e443bad344ce89ce69871406c5a29" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.631068 6579 raft_consensus.cc:3037] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.633953 6577 raft_consensus.cc:3032] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Stepping down as leader of term 8
I20240502 21:27:40.634099 6577 raft_consensus.cc:729] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:40.634253 6839 raft_consensus.cc:2396] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 21:27:40.634622 6904 leader_election.cc:302] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2; no voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720
I20240502 21:27:40.635037 9106 raft_consensus.cc:2726] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240502 21:27:40.637526 6577 consensus_queue.cc:260] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.637857 6577 raft_consensus.cc:3037] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.638195 6579 raft_consensus.cc:2396] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240502 21:27:40.640015 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:40.640941 6577 raft_consensus.cc:2396] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240502 21:27:40.641731 6904 leader_election.cc:302] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2; no voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720
I20240502 21:27:40.642081 9106 raft_consensus.cc:2726] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240502 21:27:40.643925 6969 leader_election.cc:288] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 9 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:40.645560 6968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "875f83053fee45b3bc1921b24745e9da"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:40.645752 6968 raft_consensus.cc:482] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.645848 6968 raft_consensus.cc:3037] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.630893 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09d2491d9dc4cbfb20f7adc1087b375" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.648782 8485 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:40.649153 6968 raft_consensus.cc:504] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.649483 6708 raft_consensus.cc:3037] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.650020 6968 leader_election.cc:288] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 8 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:40.650672 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875f83053fee45b3bc1921b24745e9da" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.650857 6577 raft_consensus.cc:3032] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 7 LEADER]: Stepping down as leader of term 7
I20240502 21:27:40.650954 6577 raft_consensus.cc:729] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:40.651538 6577 consensus_queue.cc:260] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.651801 6577 raft_consensus.cc:3037] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.651958 6432 catalog_manager.cc:5484] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 8 to 9, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 9 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:40.654690 6577 raft_consensus.cc:2396] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240502 21:27:40.654966 6968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "444ccd95dc0f4bdd806ca9151d56d5cd"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:40.655143 6968 raft_consensus.cc:482] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.655241 6968 raft_consensus.cc:3037] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.657428 6820 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:40.657894 6968 raft_consensus.cc:504] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.658739 6968 leader_election.cc:288] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 7 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:40.659408 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444ccd95dc0f4bdd806ca9151d56d5cd" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.659616 6840 raft_consensus.cc:3037] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.650846 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875f83053fee45b3bc1921b24745e9da" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.660792 6711 raft_consensus.cc:3037] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.660838 6968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c9e728880af44303a975a63a4bd7bc62"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:40.661068 6968 raft_consensus.cc:482] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.661168 6968 raft_consensus.cc:3037] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.662115 6840 raft_consensus.cc:2454] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 7.
I20240502 21:27:40.662524 6904 leader_election.cc:302] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:40.662825 9002 raft_consensus.cc:2781] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Leader election won for term 7
I20240502 21:27:40.662946 9002 raft_consensus.cc:686] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 7 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:40.663205 9002 consensus_queue.cc:237] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.663426 6711 raft_consensus.cc:2396] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240502 21:27:40.663645 6710 raft_consensus.cc:1261] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Refusing update from remote peer 51bc1a0939fe49348a18d3da18f9feae: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240502 21:27:40.658864 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444ccd95dc0f4bdd806ca9151d56d5cd" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.663826 6709 raft_consensus.cc:3037] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.664500 6906 leader_election.cc:302] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2; no voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:40.664989 9106 raft_consensus.cc:2726] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240502 21:27:40.665958 8965 consensus_queue.cc:1035] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:40.666134 8965 consensus_queue.cc:1225] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Peer ab5c055dc24f4944a58d7d2593ddc8ef log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240502 21:27:40.666939 6710 pending_rounds.cc:76] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef: Aborting all ops after (but not including) 3
I20240502 21:27:40.667029 6710 pending_rounds.cc:97] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20240502 21:27:40.667119 6710 raft_consensus.cc:2866] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:27:40.667215 6710 pending_rounds.cc:97] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20240502 21:27:40.667268 6710 raft_consensus.cc:2866] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:27:40.668182 6709 raft_consensus.cc:2396] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240502 21:27:40.668697 6711 raft_consensus.cc:1261] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Refusing update from remote peer 51bc1a0939fe49348a18d3da18f9feae: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240502 21:27:40.672385 6708 raft_consensus.cc:2396] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240502 21:27:40.672689 9273 consensus_queue.cc:1035] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:40.672894 9273 consensus_queue.cc:1225] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Peer ab5c055dc24f4944a58d7d2593ddc8ef log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240502 21:27:40.679366 6968 raft_consensus.cc:504] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.682063 6968 leader_election.cc:288] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 9 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:40.683070 6969 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "533d0913d0bc4582b976a58bda8176bc"
dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
from {username='slave'} at 127.0.0.1:38994
I20240502 21:27:40.683267 6969 raft_consensus.cc:482] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.683382 6969 raft_consensus.cc:3037] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.685760 6969 raft_consensus.cc:504] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.686702 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e728880af44303a975a63a4bd7bc62" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.686908 6577 raft_consensus.cc:3032] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Stepping down as leader of term 8
I20240502 21:27:40.686930 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533d0913d0bc4582b976a58bda8176bc" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.687031 6577 raft_consensus.cc:729] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:40.687162 6579 raft_consensus.cc:3032] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Stepping down as leader of term 8
I20240502 21:27:40.687230 6579 raft_consensus.cc:729] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:40.687299 6577 consensus_queue.cc:260] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.687428 6579 consensus_queue.cc:260] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.687608 6577 raft_consensus.cc:3037] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.687666 6579 raft_consensus.cc:3037] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.690555 6577 raft_consensus.cc:2396] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240502 21:27:40.691576 6579 raft_consensus.cc:2396] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240502 21:27:40.692291 6969 leader_election.cc:288] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 9 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:40.673650 6711 pending_rounds.cc:76] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef: Aborting all ops after (but not including) 3
I20240502 21:27:40.692566 6711 pending_rounds.cc:97] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20240502 21:27:40.692651 6711 raft_consensus.cc:2866] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:27:40.692761 6711 pending_rounds.cc:97] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20240502 21:27:40.692822 6711 raft_consensus.cc:2866] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:27:40.680325 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e728880af44303a975a63a4bd7bc62" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.686733 6707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533d0913d0bc4582b976a58bda8176bc" candidate_uuid: "80a838c00b544bf68ec4cc9c513212b2" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:40.694324 6707 raft_consensus.cc:3037] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.696997 6707 raft_consensus.cc:2454] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a838c00b544bf68ec4cc9c513212b2 in term 9.
I20240502 21:27:40.697572 6906 leader_election.cc:302] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2; no voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:40.698063 6906 leader_election.cc:302] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 51bc1a0939fe49348a18d3da18f9feae
I20240502 21:27:40.694325 6709 raft_consensus.cc:3037] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.698454 9106 raft_consensus.cc:2726] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240502 21:27:40.698599 9106 raft_consensus.cc:2781] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Leader election won for term 9
I20240502 21:27:40.698705 9106 raft_consensus.cc:686] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 9 LEADER]: Becoming Leader. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:40.698951 9106 consensus_queue.cc:237] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.703565 6709 raft_consensus.cc:2396] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 80a838c00b544bf68ec4cc9c513212b2 for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240502 21:27:40.704222 6906 leader_election.cc:302] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2; no voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:40.704617 9170 raft_consensus.cc:2726] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240502 21:27:40.706772 6690 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:40.722051 6436 catalog_manager.cc:5484] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 5 to 7, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 7 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:40.723707 6840 raft_consensus.cc:1261] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Refusing update from remote peer 51bc1a0939fe49348a18d3da18f9feae: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240502 21:27:40.723994 6949 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 21:27:40.724898 9124 consensus_queue.cc:1035] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:40.729768 6432 catalog_manager.cc:5484] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 reported cstate change: term changed from 8 to 9, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132). New cstate: current_term: 9 leader_uuid: "80a838c00b544bf68ec4cc9c513212b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:40.736523 8618 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:40.753746 6840 raft_consensus.cc:1261] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Refusing update from remote peer 51bc1a0939fe49348a18d3da18f9feae: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240502 21:27:40.754316 9124 consensus_queue.cc:1035] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:40.770963 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b2b3e30b94d47dd822539cb30f36f81"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:40.771173 6709 raft_consensus.cc:482] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.771299 6709 raft_consensus.cc:3037] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.773972 6709 raft_consensus.cc:504] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.775247 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2b3e30b94d47dd822539cb30f36f81" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.775449 6969 raft_consensus.cc:3032] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 LEADER]: Stepping down as leader of term 8
I20240502 21:27:40.775532 6709 leader_election.cc:288] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 9 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:40.775566 6969 raft_consensus.cc:729] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:40.775810 6969 consensus_queue.cc:260] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.776163 6969 raft_consensus.cc:3037] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.776093 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2b3e30b94d47dd822539cb30f36f81" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.776331 6840 raft_consensus.cc:3037] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.778422 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "df917e7196ed4939b75dc52930c87640"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:40.778614 6709 raft_consensus.cc:482] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.778753 6709 raft_consensus.cc:3037] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.779000 6969 raft_consensus.cc:2396] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240502 21:27:40.780027 6840 raft_consensus.cc:2396] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240502 21:27:40.780489 6644 leader_election.cc:302] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720
I20240502 21:27:40.781562 9312 raft_consensus.cc:2726] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240502 21:27:40.782852 6709 raft_consensus.cc:504] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.783459 6709 leader_election.cc:288] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 8 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:40.783926 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df917e7196ed4939b75dc52930c87640" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.784140 6579 raft_consensus.cc:3037] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.784302 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aae0bc634aa7424e935c8bc81a85099b"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:40.784495 6709 raft_consensus.cc:482] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.784600 6709 raft_consensus.cc:3037] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:40.787014 6579 raft_consensus.cc:2454] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 8.
I20240502 21:27:40.786954 6709 raft_consensus.cc:504] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.787345 6644 leader_election.cc:302] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:40.787775 9312 raft_consensus.cc:2781] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Leader election won for term 8
I20240502 21:27:40.787768 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df917e7196ed4939b75dc52930c87640" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.787968 6840 raft_consensus.cc:3037] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.788467 9312 raft_consensus.cc:686] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:40.788857 9312 consensus_queue.cc:237] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.790441 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae0bc634aa7424e935c8bc81a85099b" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 10 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.790671 6579 raft_consensus.cc:3037] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:40.790829 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae0bc634aa7424e935c8bc81a85099b" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 10 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.791020 6969 raft_consensus.cc:3032] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 9 LEADER]: Stepping down as leader of term 9
I20240502 21:27:40.791131 6969 raft_consensus.cc:729] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:40.791491 6969 consensus_queue.cc:260] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.791765 6969 raft_consensus.cc:3037] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:40.793846 6579 raft_consensus.cc:2396] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:40.794935 6969 raft_consensus.cc:2396] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:40.795372 6645 leader_election.cc:302] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:40.795667 9314 raft_consensus.cc:2726] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240502 21:27:40.795707 6709 leader_election.cc:288] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 10 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:40.796720 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:40.796892 6709 raft_consensus.cc:482] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.796991 6709 raft_consensus.cc:3037] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.798413 6840 raft_consensus.cc:2396] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20240502 21:27:40.800885 6709 raft_consensus.cc:504] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.801477 6709 leader_election.cc:288] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 9 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:40.801872 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.802073 6579 raft_consensus.cc:3032] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Stepping down as leader of term 8
I20240502 21:27:40.802142 6579 raft_consensus.cc:729] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:40.802400 6579 consensus_queue.cc:260] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.802739 6579 raft_consensus.cc:3037] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.802865 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.803071 6840 raft_consensus.cc:3037] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.805534 6579 raft_consensus.cc:2454] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 9.
I20240502 21:27:40.805850 6644 leader_election.cc:302] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:40.805922 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c10bca264dda4180bd19beb5767bf047"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:40.806139 6709 raft_consensus.cc:482] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.806164 9314 raft_consensus.cc:2781] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Leader election won for term 9
I20240502 21:27:40.806311 9314 raft_consensus.cc:686] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:40.806346 6709 raft_consensus.cc:3037] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.806598 9314 consensus_queue.cc:237] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.808113 6840 raft_consensus.cc:2454] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 9.
I20240502 21:27:40.809461 6709 raft_consensus.cc:504] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.810407 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10bca264dda4180bd19beb5767bf047" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.810595 6969 raft_consensus.cc:3037] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.813119 6969 raft_consensus.cc:2454] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 8.
I20240502 21:27:40.814666 6645 leader_election.cc:302] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:40.814937 9314 raft_consensus.cc:2781] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Leader election won for term 8
I20240502 21:27:40.815061 9314 raft_consensus.cc:686] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:40.815272 9314 consensus_queue.cc:237] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.815919 6709 leader_election.cc:288] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 8 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:40.816663 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10bca264dda4180bd19beb5767bf047" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.816856 6840 raft_consensus.cc:3032] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 7 LEADER]: Stepping down as leader of term 7
I20240502 21:27:40.816931 6840 raft_consensus.cc:729] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:40.817173 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "da14dcc58d684704b3649e85b26fb43c"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:40.817193 6840 consensus_queue.cc:260] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.817430 6709 raft_consensus.cc:482] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.817541 6840 raft_consensus.cc:3037] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.817590 6709 raft_consensus.cc:3037] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.820189 6709 raft_consensus.cc:504] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.820595 6840 raft_consensus.cc:2454] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 8.
I20240502 21:27:40.820891 6709 leader_election.cc:288] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 9 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:40.821559 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da14dcc58d684704b3649e85b26fb43c" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.821624 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da14dcc58d684704b3649e85b26fb43c" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.821753 6579 raft_consensus.cc:3032] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Stepping down as leader of term 8
I20240502 21:27:40.821791 6839 raft_consensus.cc:3037] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.821815 6579 raft_consensus.cc:729] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:40.822041 6579 consensus_queue.cc:260] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.822363 6579 raft_consensus.cc:3037] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.824923 6579 raft_consensus.cc:2454] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 9.
I20240502 21:27:40.824954 6839 raft_consensus.cc:2454] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 9.
I20240502 21:27:40.825352 6644 leader_election.cc:302] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:40.825605 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f0b941baaab542a78748d4d0894cfbf4"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:40.825796 9305 raft_consensus.cc:2781] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Leader election won for term 9
I20240502 21:27:40.825803 6709 raft_consensus.cc:482] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.825937 9305 raft_consensus.cc:686] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:40.826054 6709 raft_consensus.cc:3037] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.826225 9305 consensus_queue.cc:237] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.828943 6709 raft_consensus.cc:504] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.829886 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b941baaab542a78748d4d0894cfbf4" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 9 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.830075 6579 raft_consensus.cc:3032] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Stepping down as leader of term 8
I20240502 21:27:40.830155 6579 raft_consensus.cc:729] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:40.830404 6579 consensus_queue.cc:260] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.830628 6579 raft_consensus.cc:3037] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.837939 6579 raft_consensus.cc:2396] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240502 21:27:40.838909 6436 catalog_manager.cc:5484] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 6 to 8, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 8 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.844414 6436 catalog_manager.cc:5484] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 8 to 9, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 9 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.844987 6436 catalog_manager.cc:5484] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 8 to 9, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 9 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.845171 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b941baaab542a78748d4d0894cfbf4" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 9 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.845369 6839 raft_consensus.cc:3037] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.845418 6436 catalog_manager.cc:5484] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 7 to 8, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 8 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:40.848608 6839 raft_consensus.cc:2396] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 9 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240502 21:27:40.849118 6644 leader_election.cc:302] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720
I20240502 21:27:40.849665 9312 raft_consensus.cc:2726] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240502 21:27:40.850673 6709 leader_election.cc:288] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 9 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:40.851833 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fb023d72aa8d4248a64d4723e18a3cf6"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:40.852030 6709 raft_consensus.cc:482] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.852134 6709 raft_consensus.cc:3037] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:40.855892 6709 raft_consensus.cc:504] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.856882 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb023d72aa8d4248a64d4723e18a3cf6" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 10 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.857097 6579 raft_consensus.cc:3037] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:40.859930 6579 raft_consensus.cc:2396] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:40.860538 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb023d72aa8d4248a64d4723e18a3cf6" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 10 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.860747 6969 raft_consensus.cc:3032] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 9 LEADER]: Stepping down as leader of term 9
I20240502 21:27:40.860812 6969 raft_consensus.cc:729] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:40.861052 6969 consensus_queue.cc:260] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.861279 6969 raft_consensus.cc:3037] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:40.863668 6969 raft_consensus.cc:2396] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:40.867983 6645 leader_election.cc:302] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:40.868273 9312 raft_consensus.cc:2726] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240502 21:27:40.871059 6709 leader_election.cc:288] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 10 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:40.872416 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "980f9cc0cdc6452d8009e9f75a0d4275"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:40.872651 6709 raft_consensus.cc:482] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.872766 6709 raft_consensus.cc:3037] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.875008 6709 raft_consensus.cc:504] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.875936 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "980f9cc0cdc6452d8009e9f75a0d4275" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.875962 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "980f9cc0cdc6452d8009e9f75a0d4275" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.876133 6839 raft_consensus.cc:3037] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.876266 6969 raft_consensus.cc:3032] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 LEADER]: Stepping down as leader of term 8
I20240502 21:27:40.876374 6969 raft_consensus.cc:729] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:40.876683 6969 consensus_queue.cc:260] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.877049 6969 raft_consensus.cc:3037] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.878880 6839 raft_consensus.cc:2454] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 9.
I20240502 21:27:40.879320 6644 leader_election.cc:302] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:40.879580 9312 raft_consensus.cc:2781] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Leader election won for term 9
I20240502 21:27:40.879704 9312 raft_consensus.cc:686] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:40.879949 9312 consensus_queue.cc:237] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.883944 6969 raft_consensus.cc:2396] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240502 21:27:40.884137 6709 leader_election.cc:288] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 9 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:40.885030 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57e37f7e15b74819b692792eaed94f24"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:40.885226 6709 raft_consensus.cc:482] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.885329 6709 raft_consensus.cc:3037] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.886982 6436 catalog_manager.cc:5484] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 8 to 9, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 9 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:40.893196 6709 raft_consensus.cc:504] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.894161 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e37f7e15b74819b692792eaed94f24" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.894357 6969 raft_consensus.cc:3032] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 LEADER]: Stepping down as leader of term 7
I20240502 21:27:40.894431 6969 raft_consensus.cc:729] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:40.894670 6969 consensus_queue.cc:260] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.894920 6969 raft_consensus.cc:3037] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.897490 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e37f7e15b74819b692792eaed94f24" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.897709 6839 raft_consensus.cc:3037] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.900187 6839 raft_consensus.cc:2454] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 8.
I20240502 21:27:40.900696 6644 leader_election.cc:302] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:40.900964 9312 raft_consensus.cc:2781] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Leader election won for term 8
I20240502 21:27:40.901115 9312 raft_consensus.cc:686] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:40.901337 9312 consensus_queue.cc:237] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.900951 6709 leader_election.cc:288] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 8 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:40.908910 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "331bcde3e6bf43c081af836c8cdebdc7"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:40.909741 6969 raft_consensus.cc:2396] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240502 21:27:40.910055 6709 raft_consensus.cc:482] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.910364 6709 raft_consensus.cc:3037] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.915025 6709 raft_consensus.cc:504] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.915680 6710 raft_consensus.cc:1226] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Rejecting Update request from peer 80a838c00b544bf68ec4cc9c513212b2 for earlier term 8. Current term is 9. Ops: []
I20240502 21:27:40.916438 9106 consensus_queue.cc:1046] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:40.916741 9106 raft_consensus.cc:3032] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 LEADER]: Stepping down as leader of term 8
I20240502 21:27:40.916815 9106 raft_consensus.cc:729] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:40.917095 9106 consensus_queue.cc:260] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.917534 9106 raft_consensus.cc:3037] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.917496 6436 catalog_manager.cc:5484] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 7 to 8, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 8 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.917809 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331bcde3e6bf43c081af836c8cdebdc7" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 9 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.918097 6709 leader_election.cc:288] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 9 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:40.918943 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c6d11cba6ca04c17a9f47506824d9aa2"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:40.919116 6709 raft_consensus.cc:482] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.919200 6709 raft_consensus.cc:3037] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.920682 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331bcde3e6bf43c081af836c8cdebdc7" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 9 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.920872 6579 raft_consensus.cc:3037] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.921136 6969 raft_consensus.cc:2396] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:27:40.921360 6709 raft_consensus.cc:504] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.922515 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d11cba6ca04c17a9f47506824d9aa2" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.922766 6839 raft_consensus.cc:3037] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.923655 6579 raft_consensus.cc:2396] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:27:40.925473 6839 raft_consensus.cc:2396] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240502 21:27:40.926237 6644 leader_election.cc:302] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:40.926501 9312 raft_consensus.cc:2726] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240502 21:27:40.927379 6709 leader_election.cc:288] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 9 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:40.927845 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d11cba6ca04c17a9f47506824d9aa2" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.928069 6579 raft_consensus.cc:3037] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.928575 6710 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "444ccd95dc0f4bdd806ca9151d56d5cd"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:40.928784 6710 raft_consensus.cc:482] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.928905 6710 raft_consensus.cc:3037] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.930747 6579 raft_consensus.cc:2454] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 9.
I20240502 21:27:40.931077 6644 leader_election.cc:302] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 9793130a8e4f45388357b8d2e9f47720
I20240502 21:27:40.931399 9312 raft_consensus.cc:2781] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Leader election won for term 9
I20240502 21:27:40.931349 6710 raft_consensus.cc:504] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.931535 9312 raft_consensus.cc:686] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:40.931753 9312 consensus_queue.cc:237] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.932214 6710 leader_election.cc:288] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 8 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:40.932332 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444ccd95dc0f4bdd806ca9151d56d5cd" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.932535 6969 raft_consensus.cc:3032] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 7 LEADER]: Stepping down as leader of term 7
I20240502 21:27:40.932615 6969 raft_consensus.cc:729] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:40.932850 6969 consensus_queue.cc:260] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.933003 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444ccd95dc0f4bdd806ca9151d56d5cd" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.933106 6969 raft_consensus.cc:3037] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.933204 6839 raft_consensus.cc:3037] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.934957 6839 raft_consensus.cc:2454] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 8.
I20240502 21:27:40.936079 6969 raft_consensus.cc:2396] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240502 21:27:40.936601 6644 leader_election.cc:302] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:40.936873 9312 raft_consensus.cc:2781] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Leader election won for term 8
I20240502 21:27:40.937070 9312 raft_consensus.cc:686] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:40.937369 9312 consensus_queue.cc:237] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.937536 6436 catalog_manager.cc:5484] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 7 to 9, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 9 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:40.938882 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3093485f30b41939a1380ab7aa9e38b"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:40.939034 6709 raft_consensus.cc:482] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.939159 6709 raft_consensus.cc:3037] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.941045 6709 raft_consensus.cc:504] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.941567 6709 leader_election.cc:288] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 9 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:40.941998 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3093485f30b41939a1380ab7aa9e38b" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.942346 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3093485f30b41939a1380ab7aa9e38b" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.942230 6969 raft_consensus.cc:3037] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.942498 6839 raft_consensus.cc:3037] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.942790 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1c98cbfbd0a44335b75b228c217f9132"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:40.942965 6709 raft_consensus.cc:482] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.943053 6709 raft_consensus.cc:3037] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.944581 6839 raft_consensus.cc:2396] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240502 21:27:40.944964 6969 raft_consensus.cc:2454] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 9.
I20240502 21:27:40.945304 6645 leader_election.cc:302] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 9793130a8e4f45388357b8d2e9f47720
I20240502 21:27:40.945374 6709 raft_consensus.cc:504] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.945677 9312 raft_consensus.cc:2781] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Leader election won for term 9
I20240502 21:27:40.945890 9312 raft_consensus.cc:686] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:40.945948 6709 leader_election.cc:288] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 7 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:40.945890 6436 catalog_manager.cc:5484] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 7 to 8, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 8 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.946121 9312 consensus_queue.cc:237] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.946424 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c98cbfbd0a44335b75b228c217f9132" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.946612 6579 raft_consensus.cc:3037] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.948186 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c98cbfbd0a44335b75b228c217f9132" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.948396 6839 raft_consensus.cc:3037] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [term 6 FOLLOWER]: Advancing to term 7
I20240502 21:27:40.949373 6579 raft_consensus.cc:2454] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 7.
I20240502 21:27:40.949702 6644 leader_election.cc:302] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:40.949950 9312 raft_consensus.cc:2781] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Leader election won for term 7
I20240502 21:27:40.950233 9312 raft_consensus.cc:686] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:40.950178 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ea3b52c673ae4b9f931059c4b0a456fa"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:40.950450 6709 raft_consensus.cc:482] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.950579 6839 raft_consensus.cc:2396] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20240502 21:27:40.950579 6709 raft_consensus.cc:3037] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.950765 9312 consensus_queue.cc:237] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.952708 6709 raft_consensus.cc:504] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.953218 6709 leader_election.cc:288] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 9 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:40.953612 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3b52c673ae4b9f931059c4b0a456fa" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.953792 6969 raft_consensus.cc:3032] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 8 LEADER]: Stepping down as leader of term 8
I20240502 21:27:40.953881 6969 raft_consensus.cc:729] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:40.954108 6969 consensus_queue.cc:260] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.954375 6969 raft_consensus.cc:3037] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.955546 6436 catalog_manager.cc:5484] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 7 to 9, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 9 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.956033 6436 catalog_manager.cc:5484] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 5 to 7, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 7 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.956879 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3b52c673ae4b9f931059c4b0a456fa" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.956981 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "169899da82594d9cb673f8db74764ca2"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:40.957060 6839 raft_consensus.cc:3037] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.957190 6709 raft_consensus.cc:482] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.957311 6709 raft_consensus.cc:3037] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.959002 6839 raft_consensus.cc:2454] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 9.
I20240502 21:27:40.959416 6644 leader_election.cc:302] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:40.959751 9312 raft_consensus.cc:2781] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Leader election won for term 9
I20240502 21:27:40.959911 9312 raft_consensus.cc:686] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:40.959859 6709 raft_consensus.cc:504] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.960141 9312 consensus_queue.cc:237] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.960721 6969 raft_consensus.cc:2396] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240502 21:27:40.960814 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169899da82594d9cb673f8db74764ca2" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.961004 6579 raft_consensus.cc:3037] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.962953 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169899da82594d9cb673f8db74764ca2" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.963124 6839 raft_consensus.cc:3037] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.963877 6579 raft_consensus.cc:2454] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 9.
I20240502 21:27:40.964103 6709 leader_election.cc:288] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 9 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:40.964962 6644 leader_election.cc:302] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:40.965101 6839 raft_consensus.cc:2396] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240502 21:27:40.965497 9312 raft_consensus.cc:2781] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Leader election won for term 9
I20240502 21:27:40.965677 9312 raft_consensus.cc:686] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:40.965746 6436 catalog_manager.cc:5484] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 8 to 9, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 9 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.965975 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a09d2491d9dc4cbfb20f7adc1087b375"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:40.965984 9312 consensus_queue.cc:237] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:40.966145 6709 raft_consensus.cc:482] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.966310 6709 raft_consensus.cc:3037] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.968609 6709 raft_consensus.cc:504] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.969446 6709 leader_election.cc:288] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 9 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:40.969553 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09d2491d9dc4cbfb20f7adc1087b375" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.969614 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09d2491d9dc4cbfb20f7adc1087b375" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.969712 6839 raft_consensus.cc:3037] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.969828 6969 raft_consensus.cc:3037] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.972108 6969 raft_consensus.cc:2454] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 9.
I20240502 21:27:40.971992 6436 catalog_manager.cc:5484] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 7 to 9, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 9 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.972718 6645 leader_election.cc:302] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:40.972884 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8fac59b25c164e658fe71cfdf7ca78cb"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:40.972970 9312 raft_consensus.cc:2781] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Leader election won for term 9
I20240502 21:27:40.973066 6709 raft_consensus.cc:482] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.973167 6709 raft_consensus.cc:3037] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.973197 9312 raft_consensus.cc:686] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:40.973423 9312 consensus_queue.cc:237] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.973942 6839 raft_consensus.cc:2396] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240502 21:27:40.975808 6709 raft_consensus.cc:504] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.976454 6709 leader_election.cc:288] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 8 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:40.976866 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fac59b25c164e658fe71cfdf7ca78cb" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:40.976886 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fac59b25c164e658fe71cfdf7ca78cb" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.977046 6579 raft_consensus.cc:3037] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.977102 6839 raft_consensus.cc:3037] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:40.979612 6579 raft_consensus.cc:2454] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 8.
I20240502 21:27:40.979660 6839 raft_consensus.cc:2396] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240502 21:27:40.980111 6644 leader_election.cc:302] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 9793130a8e4f45388357b8d2e9f47720
I20240502 21:27:40.980392 9312 raft_consensus.cc:2781] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Leader election won for term 8
I20240502 21:27:40.980659 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1325bd7bbb4340f88bdc7c705934862e"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:40.980813 6709 raft_consensus.cc:482] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.980926 6709 raft_consensus.cc:3037] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:40.981359 6436 catalog_manager.cc:5484] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 7 to 9, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 9 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:40.984014 6709 raft_consensus.cc:504] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.984535 9312 raft_consensus.cc:686] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:40.984805 6709 leader_election.cc:288] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 10 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:40.985222 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1325bd7bbb4340f88bdc7c705934862e" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 10 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.985407 6969 raft_consensus.cc:3032] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 9 LEADER]: Stepping down as leader of term 9
I20240502 21:27:40.985484 6969 raft_consensus.cc:729] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:40.985507 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1325bd7bbb4340f88bdc7c705934862e" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 10 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.984850 9312 consensus_queue.cc:237] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:40.985746 6579 raft_consensus.cc:3037] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:40.985724 6969 consensus_queue.cc:260] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:40.985982 6969 raft_consensus.cc:3037] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:40.989356 6969 raft_consensus.cc:2396] T 1325bd7bbb4340f88bdc7c705934862e P 80a838c00b544bf68ec4cc9c513212b2 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:40.989929 6579 raft_consensus.cc:2396] T 1325bd7bbb4340f88bdc7c705934862e P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:40.990478 6644 leader_election.cc:302] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:40.990830 9314 raft_consensus.cc:2726] T 1325bd7bbb4340f88bdc7c705934862e P ab5c055dc24f4944a58d7d2593ddc8ef [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240502 21:27:40.991055 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "875f83053fee45b3bc1921b24745e9da"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:40.991291 6709 raft_consensus.cc:482] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:40.991432 6709 raft_consensus.cc:3037] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.994362 6436 catalog_manager.cc:5484] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 6 to 8, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 8 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:40.994719 6709 raft_consensus.cc:504] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:40.995463 6709 leader_election.cc:288] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 9 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:40.996016 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875f83053fee45b3bc1921b24745e9da" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:40.996117 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875f83053fee45b3bc1921b24745e9da" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:40.996258 6969 raft_consensus.cc:3037] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.996268 6579 raft_consensus.cc:3037] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:40.998476 6579 raft_consensus.cc:2396] T 875f83053fee45b3bc1921b24745e9da P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240502 21:27:40.999574 6969 raft_consensus.cc:2454] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 9.
I20240502 21:27:40.999953 6645 leader_election.cc:302] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 51bc1a0939fe49348a18d3da18f9feae
I20240502 21:27:41.000259 9312 raft_consensus.cc:2781] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Leader election won for term 9
I20240502 21:27:41.000453 9312 raft_consensus.cc:686] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:41.000552 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "533d0913d0bc4582b976a58bda8176bc"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:41.000723 6709 raft_consensus.cc:482] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.000686 9312 consensus_queue.cc:237] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:41.000869 6709 raft_consensus.cc:3037] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.003715 6436 catalog_manager.cc:5484] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 7 to 9, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 9 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:41.005052 6709 raft_consensus.cc:504] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:41.006026 6709 leader_election.cc:288] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 10 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:41.006203 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533d0913d0bc4582b976a58bda8176bc" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 10 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:41.006731 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533d0913d0bc4582b976a58bda8176bc" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 10 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:41.006920 6969 raft_consensus.cc:3032] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 9 LEADER]: Stepping down as leader of term 9
I20240502 21:27:41.006923 6579 raft_consensus.cc:3037] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.007038 6969 raft_consensus.cc:729] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 80a838c00b544bf68ec4cc9c513212b2, State: Running, Role: LEADER
I20240502 21:27:41.007263 6969 consensus_queue.cc:260] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:41.007484 6969 raft_consensus.cc:3037] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.009032 6579 raft_consensus.cc:2396] T 533d0913d0bc4582b976a58bda8176bc P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:41.009780 6969 raft_consensus.cc:2396] T 533d0913d0bc4582b976a58bda8176bc P 80a838c00b544bf68ec4cc9c513212b2 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:41.010152 6645 leader_election.cc:302] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:41.010181 6709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c9e728880af44303a975a63a4bd7bc62"
dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
from {username='slave'} at 127.0.0.1:35174
I20240502 21:27:41.010399 6709 raft_consensus.cc:482] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.010459 9312 raft_consensus.cc:2726] T 533d0913d0bc4582b976a58bda8176bc P ab5c055dc24f4944a58d7d2593ddc8ef [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240502 21:27:41.010540 6709 raft_consensus.cc:3037] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.013298 6709 raft_consensus.cc:504] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:41.013833 6709 leader_election.cc:288] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 10 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:41.014247 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e728880af44303a975a63a4bd7bc62" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:41.014282 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e728880af44303a975a63a4bd7bc62" candidate_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:41.014393 6969 raft_consensus.cc:3037] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.014451 6579 raft_consensus.cc:3037] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.016520 6579 raft_consensus.cc:2396] T c9e728880af44303a975a63a4bd7bc62 P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5c055dc24f4944a58d7d2593ddc8ef for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20240502 21:27:41.016494 6969 raft_consensus.cc:2454] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5c055dc24f4944a58d7d2593ddc8ef in term 10.
I20240502 21:27:41.016973 6645 leader_election.cc:302] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 51bc1a0939fe49348a18d3da18f9feae
I20240502 21:27:41.017683 9312 raft_consensus.cc:2781] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [term 10 FOLLOWER]: Leader election won for term 10
I20240502 21:27:41.017869 9312 raft_consensus.cc:686] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [term 10 LEADER]: Becoming Leader. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:41.018160 9312 consensus_queue.cc:237] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:41.021499 6436 catalog_manager.cc:5484] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef reported cstate change: term changed from 8 to 10, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130). New cstate: current_term: 10 leader_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:41.031374 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b2b3e30b94d47dd822539cb30f36f81"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:41.031580 6839 raft_consensus.cc:482] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.031708 6839 raft_consensus.cc:3037] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.033675 6839 raft_consensus.cc:504] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:41.034276 6839 leader_election.cc:288] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 10 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:41.034783 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2b3e30b94d47dd822539cb30f36f81" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:41.035027 6969 raft_consensus.cc:3037] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.035111 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2b3e30b94d47dd822539cb30f36f81" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:41.035315 6709 raft_consensus.cc:3037] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.035480 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "df917e7196ed4939b75dc52930c87640"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:41.035665 6839 raft_consensus.cc:482] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.035749 6839 raft_consensus.cc:3037] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:41.038275 6969 raft_consensus.cc:2396] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20240502 21:27:41.038822 6839 raft_consensus.cc:504] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:41.039796 6839 leader_election.cc:288] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 9 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:41.040047 6710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df917e7196ed4939b75dc52930c87640" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:41.040135 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df917e7196ed4939b75dc52930c87640" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:41.040266 6710 raft_consensus.cc:3032] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 LEADER]: Stepping down as leader of term 8
I20240502 21:27:41.040350 6710 raft_consensus.cc:729] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:41.040386 6579 raft_consensus.cc:3037] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:41.040656 6710 consensus_queue.cc:260] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:41.040859 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3619e8a307de4583a4e470ad583fd7bc"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:41.040985 6710 raft_consensus.cc:3037] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:41.041024 6839 raft_consensus.cc:482] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.041149 6839 raft_consensus.cc:3037] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.043226 6710 raft_consensus.cc:2396] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240502 21:27:41.043172 6579 raft_consensus.cc:2454] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 9.
I20240502 21:27:41.043715 6774 leader_election.cc:302] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:41.043711 6839 raft_consensus.cc:504] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:41.044026 9248 raft_consensus.cc:2781] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Leader election won for term 9
I20240502 21:27:41.044163 9248 raft_consensus.cc:686] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 9 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:41.044390 9248 consensus_queue.cc:237] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:41.045125 6839 leader_election.cc:288] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 10 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:41.046586 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3619e8a307de4583a4e470ad583fd7bc" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:41.046766 6579 raft_consensus.cc:3037] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.051167 6579 raft_consensus.cc:2396] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20240502 21:27:41.051190 6436 catalog_manager.cc:5484] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 8 to 9, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 9 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:41.051648 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:41.052048 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3619e8a307de4583a4e470ad583fd7bc" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:41.053228 6839 raft_consensus.cc:482] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.053228 6709 raft_consensus.cc:2454] T 6b2b3e30b94d47dd822539cb30f36f81 P ab5c055dc24f4944a58d7d2593ddc8ef [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 10.
I20240502 21:27:41.053409 6839 raft_consensus.cc:3037] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.053848 6969 raft_consensus.cc:3037] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.053601 6776 leader_election.cc:302] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 80a838c00b544bf68ec4cc9c513212b2
I20240502 21:27:41.054512 9248 raft_consensus.cc:2781] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Leader election won for term 10
I20240502 21:27:41.054663 9248 raft_consensus.cc:686] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [term 10 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:41.054919 9248 consensus_queue.cc:237] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:41.059386 6839 raft_consensus.cc:504] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:41.059922 6436 catalog_manager.cc:5484] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 8 to 10, leader changed from 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 10 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:41.060382 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:41.060407 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:41.060631 6579 raft_consensus.cc:3037] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.060792 6709 raft_consensus.cc:3032] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 LEADER]: Stepping down as leader of term 9
I20240502 21:27:41.060922 6709 raft_consensus.cc:729] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:41.061230 6709 consensus_queue.cc:260] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:41.061491 6709 raft_consensus.cc:3037] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.063946 6579 raft_consensus.cc:2454] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 10.
I20240502 21:27:41.063992 6709 raft_consensus.cc:2396] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20240502 21:27:41.064340 6774 leader_election.cc:302] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:41.064833 9248 raft_consensus.cc:2781] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Leader election won for term 10
I20240502 21:27:41.064854 6969 raft_consensus.cc:2454] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 10.
I20240502 21:27:41.064990 9248 raft_consensus.cc:686] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 10 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:41.065178 6839 leader_election.cc:288] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 10 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:41.065235 9248 consensus_queue.cc:237] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:41.066475 6775 leader_election.cc:302] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters: 51bc1a0939fe49348a18d3da18f9feae
I20240502 21:27:41.066854 9077 raft_consensus.cc:2781] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Leader election won for term 10
I20240502 21:27:41.066999 9077 raft_consensus.cc:686] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 10 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:41.067222 9077 consensus_queue.cc:237] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:41.070176 6436 catalog_manager.cc:5484] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 9 to 10, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 10 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:41.072610 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c10bca264dda4180bd19beb5767bf047"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:41.072860 6839 raft_consensus.cc:482] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.073107 6839 raft_consensus.cc:3037] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:41.074967 6436 catalog_manager.cc:5484] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 8 to 10, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 10 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:41.076911 6839 raft_consensus.cc:504] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:41.077950 6839 leader_election.cc:288] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 9 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:41.077971 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10bca264dda4180bd19beb5767bf047" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:41.078188 6969 raft_consensus.cc:3037] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:41.079224 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10bca264dda4180bd19beb5767bf047" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:41.079432 6709 raft_consensus.cc:3032] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 LEADER]: Stepping down as leader of term 8
I20240502 21:27:41.079524 6709 raft_consensus.cc:729] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:41.079761 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "66198d766e6e4ceb820bc92bcca67d85"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:41.079795 6709 consensus_queue.cc:260] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:41.079948 6839 raft_consensus.cc:482] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.080039 6839 raft_consensus.cc:3037] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.080157 6709 raft_consensus.cc:3037] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:41.082190 6969 raft_consensus.cc:2454] T c10bca264dda4180bd19beb5767bf047 P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 9.
I20240502 21:27:41.082569 6775 leader_election.cc:302] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:41.082829 9077 raft_consensus.cc:2781] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Leader election won for term 9
I20240502 21:27:41.082931 9077 raft_consensus.cc:686] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [term 9 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:41.083014 6709 raft_consensus.cc:2396] T c10bca264dda4180bd19beb5767bf047 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20240502 21:27:41.083166 9077 consensus_queue.cc:237] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:41.086333 6436 catalog_manager.cc:5484] T c10bca264dda4180bd19beb5767bf047 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 8 to 9, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 9 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:41.087266 6839 raft_consensus.cc:504] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:41.088174 6839 leader_election.cc:288] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 10 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:41.090104 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "980f9cc0cdc6452d8009e9f75a0d4275"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:41.090273 6839 raft_consensus.cc:482] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.090389 6839 raft_consensus.cc:3037] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.090312 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66198d766e6e4ceb820bc92bcca67d85" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:41.090503 6579 raft_consensus.cc:3037] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.093425 6579 raft_consensus.cc:2396] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20240502 21:27:41.093824 6839 raft_consensus.cc:504] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:41.094051 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66198d766e6e4ceb820bc92bcca67d85" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:41.094331 6969 raft_consensus.cc:3037] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.094686 6839 leader_election.cc:288] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 10 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:41.094980 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "980f9cc0cdc6452d8009e9f75a0d4275" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 10 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:41.095180 6968 raft_consensus.cc:3037] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.095257 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "980f9cc0cdc6452d8009e9f75a0d4275" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 10 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:41.095510 6709 raft_consensus.cc:3032] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 LEADER]: Stepping down as leader of term 9
I20240502 21:27:41.095599 6709 raft_consensus.cc:729] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:41.095731 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f0b941baaab542a78748d4d0894cfbf4"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:41.095953 6839 raft_consensus.cc:482] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.096134 6839 raft_consensus.cc:3037] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.095953 6709 consensus_queue.cc:260] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:41.096323 6709 raft_consensus.cc:3037] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.098037 6969 raft_consensus.cc:2454] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 10.
I20240502 21:27:41.098577 6775 leader_election.cc:302] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters: 51bc1a0939fe49348a18d3da18f9feae
I20240502 21:27:41.099143 9248 raft_consensus.cc:2781] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Leader election won for term 10
I20240502 21:27:41.099288 9248 raft_consensus.cc:686] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 10 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:41.099170 6839 raft_consensus.cc:504] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:41.099464 6709 raft_consensus.cc:2396] T 980f9cc0cdc6452d8009e9f75a0d4275 P ab5c055dc24f4944a58d7d2593ddc8ef [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:41.099757 6968 raft_consensus.cc:2396] T 980f9cc0cdc6452d8009e9f75a0d4275 P 80a838c00b544bf68ec4cc9c513212b2 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:41.100320 9248 consensus_queue.cc:237] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:41.100595 6775 leader_election.cc:302] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720; no voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:41.100914 6839 leader_election.cc:288] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 10 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:41.101634 9077 raft_consensus.cc:2726] T 980f9cc0cdc6452d8009e9f75a0d4275 P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240502 21:27:41.101703 6710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b941baaab542a78748d4d0894cfbf4" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:41.101904 6710 raft_consensus.cc:3037] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.102111 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3b6b526ec34c41aea64a2a19f0208f5d"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:41.102272 6839 raft_consensus.cc:482] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.102388 6839 raft_consensus.cc:3037] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:41.102584 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b941baaab542a78748d4d0894cfbf4" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:41.102775 6579 raft_consensus.cc:3037] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.104741 6710 raft_consensus.cc:2454] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 10.
I20240502 21:27:41.105019 6839 raft_consensus.cc:504] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:41.105273 6776 leader_election.cc:302] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:41.105576 9077 raft_consensus.cc:2781] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Leader election won for term 10
I20240502 21:27:41.105859 6839 leader_election.cc:288] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 9 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:41.106359 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6b526ec34c41aea64a2a19f0208f5d" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:41.106515 6577 raft_consensus.cc:3037] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:41.106832 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "da14dcc58d684704b3649e85b26fb43c"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:41.107039 6839 raft_consensus.cc:482] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.107154 6839 raft_consensus.cc:3037] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.109242 6577 raft_consensus.cc:2396] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240502 21:27:41.109678 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6b526ec34c41aea64a2a19f0208f5d" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:41.109871 6969 raft_consensus.cc:3037] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:41.110595 6839 raft_consensus.cc:504] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:41.111315 6839 leader_election.cc:288] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 10 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:41.111752 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da14dcc58d684704b3649e85b26fb43c" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:41.111758 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da14dcc58d684704b3649e85b26fb43c" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:41.111971 6709 raft_consensus.cc:3032] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 LEADER]: Stepping down as leader of term 9
I20240502 21:27:41.112001 6577 raft_consensus.cc:3037] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.112068 6709 raft_consensus.cc:729] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:41.112180 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5e38d544ee14490c9fb5d8069440fd6a"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:41.112346 6839 raft_consensus.cc:482] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.112531 6839 raft_consensus.cc:3037] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.112452 6709 consensus_queue.cc:260] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:41.112671 6969 raft_consensus.cc:2454] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 9.
I20240502 21:27:41.112790 6709 raft_consensus.cc:3037] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.113109 6775 leader_election.cc:302] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters: 51bc1a0939fe49348a18d3da18f9feae
I20240502 21:27:41.115031 6839 raft_consensus.cc:504] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:41.115456 6709 raft_consensus.cc:2396] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20240502 21:27:41.115674 6839 leader_election.cc:288] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 10 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:41.115912 6580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e38d544ee14490c9fb5d8069440fd6a" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:41.116075 6580 raft_consensus.cc:3037] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.118682 6580 raft_consensus.cc:2396] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20240502 21:27:41.125051 6577 raft_consensus.cc:2454] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 10.
I20240502 21:27:41.125470 6774 leader_election.cc:302] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters: ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:41.135845 9359 raft_consensus.cc:2781] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Leader election won for term 9
I20240502 21:27:41.136181 9359 raft_consensus.cc:686] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 9 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:41.136550 9359 consensus_queue.cc:237] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:41.141323 6579 raft_consensus.cc:2396] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 10 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20240502 21:27:41.147485 9359 raft_consensus.cc:2781] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Leader election won for term 10
I20240502 21:27:41.148211 9359 raft_consensus.cc:686] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 10 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:41.148496 9359 consensus_queue.cc:237] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:41.152616 9077 raft_consensus.cc:686] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 10 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:41.152851 9077 consensus_queue.cc:237] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:41.154717 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e38d544ee14490c9fb5d8069440fd6a" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:41.154933 6969 raft_consensus.cc:3037] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.155419 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a09d2491d9dc4cbfb20f7adc1087b375"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:41.155599 6839 raft_consensus.cc:482] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.155687 6839 raft_consensus.cc:3037] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.157297 6439 catalog_manager.cc:5484] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 8 to 10, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 10 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:41.158042 6839 raft_consensus.cc:504] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:41.158335 6969 raft_consensus.cc:2454] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 10.
I20240502 21:27:41.160086 6839 leader_election.cc:288] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 10 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:41.160835 6775 leader_election.cc:302] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters: 51bc1a0939fe49348a18d3da18f9feae
I20240502 21:27:41.160825 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09d2491d9dc4cbfb20f7adc1087b375" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:41.161031 6709 raft_consensus.cc:3032] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 LEADER]: Stepping down as leader of term 9
I20240502 21:27:41.161113 6709 raft_consensus.cc:729] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:41.161216 9248 raft_consensus.cc:2781] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Leader election won for term 10
I20240502 21:27:41.161334 9248 raft_consensus.cc:686] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 10 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:41.161377 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09d2491d9dc4cbfb20f7adc1087b375" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:41.161388 6709 consensus_queue.cc:260] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:41.161597 6969 raft_consensus.cc:3037] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.161530 9248 consensus_queue.cc:237] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:41.161638 6709 raft_consensus.cc:3037] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.164682 6969 raft_consensus.cc:2454] T a09d2491d9dc4cbfb20f7adc1087b375 P 80a838c00b544bf68ec4cc9c513212b2 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 10.
I20240502 21:27:41.165139 6775 leader_election.cc:302] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:41.165555 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5b2e443bad344ce89ce69871406c5a29"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:41.165699 6839 raft_consensus.cc:482] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.165800 6839 raft_consensus.cc:3037] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.166168 9354 raft_consensus.cc:2781] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Leader election won for term 10
I20240502 21:27:41.166332 9354 raft_consensus.cc:686] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [term 10 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:41.166772 9354 consensus_queue.cc:237] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:41.169026 6839 raft_consensus.cc:504] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:41.169649 6839 leader_election.cc:288] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 10 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:41.170672 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2e443bad344ce89ce69871406c5a29" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:41.170876 6969 raft_consensus.cc:3037] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.171308 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2e443bad344ce89ce69871406c5a29" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:41.171490 6579 raft_consensus.cc:3037] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.171936 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57e37f7e15b74819b692792eaed94f24"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:41.172116 6839 raft_consensus.cc:482] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.172214 6839 raft_consensus.cc:3037] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:41.173382 6709 raft_consensus.cc:2396] T a09d2491d9dc4cbfb20f7adc1087b375 P ab5c055dc24f4944a58d7d2593ddc8ef [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20240502 21:27:41.173624 6969 raft_consensus.cc:2454] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 10.
I20240502 21:27:41.174252 6579 raft_consensus.cc:2396] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20240502 21:27:41.174531 6775 leader_election.cc:302] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:41.175243 6839 raft_consensus.cc:504] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:41.175637 9353 raft_consensus.cc:2781] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Leader election won for term 10
I20240502 21:27:41.175957 9353 raft_consensus.cc:686] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 10 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:41.176136 6839 leader_election.cc:288] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 9 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:41.176455 9353 consensus_queue.cc:237] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:41.176695 6710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e37f7e15b74819b692792eaed94f24" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 9 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:41.176928 6710 raft_consensus.cc:3032] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 LEADER]: Stepping down as leader of term 8
I20240502 21:27:41.177008 6710 raft_consensus.cc:729] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:41.177290 6710 consensus_queue.cc:260] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:41.177625 6710 raft_consensus.cc:3037] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:41.177821 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e37f7e15b74819b692792eaed94f24" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 9 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:41.178053 6969 raft_consensus.cc:3037] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:41.181231 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fbec2b1b80644a93b9239600157ad957"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:41.181362 6969 raft_consensus.cc:2396] T 57e37f7e15b74819b692792eaed94f24 P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:27:41.181392 6839 raft_consensus.cc:482] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.181497 6839 raft_consensus.cc:3037] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:41.183215 6710 raft_consensus.cc:2396] T 57e37f7e15b74819b692792eaed94f24 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:27:41.183694 6776 leader_election.cc:302] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720; no voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:41.183960 9354 raft_consensus.cc:2726] T 57e37f7e15b74819b692792eaed94f24 P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240502 21:27:41.184021 6839 raft_consensus.cc:504] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:41.184964 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec2b1b80644a93b9239600157ad957" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:41.184958 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec2b1b80644a93b9239600157ad957" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:41.185180 6969 raft_consensus.cc:3037] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:41.185178 6579 raft_consensus.cc:3037] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:41.187175 6839 leader_election.cc:288] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 9 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:41.187688 6969 raft_consensus.cc:2454] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 9.
I20240502 21:27:41.187831 6579 raft_consensus.cc:2396] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240502 21:27:41.188095 6775 leader_election.cc:302] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:41.188670 9354 raft_consensus.cc:2781] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Leader election won for term 9
I20240502 21:27:41.188912 9354 raft_consensus.cc:686] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [term 9 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:41.188978 6839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c6d11cba6ca04c17a9f47506824d9aa2"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:41.189260 6839 raft_consensus.cc:482] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.189385 9354 consensus_queue.cc:237] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:41.189846 6839 raft_consensus.cc:3037] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.197798 6839 raft_consensus.cc:504] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:41.198429 6839 leader_election.cc:288] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 10 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:41.199172 6710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d11cba6ca04c17a9f47506824d9aa2" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:41.199141 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d11cba6ca04c17a9f47506824d9aa2" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:41.199293 6840 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "444ccd95dc0f4bdd806ca9151d56d5cd"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:41.199374 6710 raft_consensus.cc:3032] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 LEADER]: Stepping down as leader of term 9
I20240502 21:27:41.199371 6579 raft_consensus.cc:3037] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.199450 6710 raft_consensus.cc:729] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:41.199460 6840 raft_consensus.cc:482] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.199545 6840 raft_consensus.cc:3037] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:41.199735 6710 consensus_queue.cc:260] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:41.199990 6710 raft_consensus.cc:3037] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.201819 6840 raft_consensus.cc:504] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:41.202383 6840 leader_election.cc:288] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 9 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:41.202785 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444ccd95dc0f4bdd806ca9151d56d5cd" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 9 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:41.202947 6709 raft_consensus.cc:3032] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 LEADER]: Stepping down as leader of term 8
I20240502 21:27:41.203028 6709 raft_consensus.cc:729] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:41.203116 6579 raft_consensus.cc:2454] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 10.
I20240502 21:27:41.203296 6709 consensus_queue.cc:260] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:41.203694 6774 leader_election.cc:302] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:41.203923 6709 raft_consensus.cc:3037] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:41.204119 6840 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3093485f30b41939a1380ab7aa9e38b"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:41.204301 6840 raft_consensus.cc:482] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.204450 6840 raft_consensus.cc:3037] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.204315 6969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444ccd95dc0f4bdd806ca9151d56d5cd" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 9 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:41.204692 6969 raft_consensus.cc:3037] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:41.206930 6840 raft_consensus.cc:504] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:41.207646 6840 leader_election.cc:288] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 10 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:41.207863 9354 raft_consensus.cc:2781] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Leader election won for term 10
I20240502 21:27:41.208086 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3093485f30b41939a1380ab7aa9e38b" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:41.208261 6711 raft_consensus.cc:3032] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 LEADER]: Stepping down as leader of term 9
I20240502 21:27:41.208369 6711 raft_consensus.cc:729] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:41.208477 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3093485f30b41939a1380ab7aa9e38b" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:41.208700 6968 raft_consensus.cc:3037] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.208643 6711 consensus_queue.cc:260] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:41.208834 6969 raft_consensus.cc:2396] T 444ccd95dc0f4bdd806ca9151d56d5cd P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:27:41.208959 6711 raft_consensus.cc:3037] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.209147 9354 raft_consensus.cc:686] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 10 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:41.209489 9354 consensus_queue.cc:237] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:41.211257 6968 raft_consensus.cc:2454] T d3093485f30b41939a1380ab7aa9e38b P 80a838c00b544bf68ec4cc9c513212b2 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 10.
I20240502 21:27:41.211445 6711 raft_consensus.cc:2396] T d3093485f30b41939a1380ab7aa9e38b P ab5c055dc24f4944a58d7d2593ddc8ef [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20240502 21:27:41.212200 6775 leader_election.cc:302] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters: ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:41.212496 9353 raft_consensus.cc:2781] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Leader election won for term 10
I20240502 21:27:41.212636 9353 raft_consensus.cc:686] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [term 10 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:41.212846 9353 consensus_queue.cc:237] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:41.213655 6709 raft_consensus.cc:2396] T 444ccd95dc0f4bdd806ca9151d56d5cd P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 21:27:41.214751 6710 raft_consensus.cc:2396] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20240502 21:27:41.215140 6776 leader_election.cc:302] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720; no voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:41.215689 9359 raft_consensus.cc:2726] T 444ccd95dc0f4bdd806ca9151d56d5cd P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240502 21:27:41.218432 6840 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1c98cbfbd0a44335b75b228c217f9132"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:41.218636 6840 raft_consensus.cc:482] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.218794 6840 raft_consensus.cc:3037] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:41.221943 6840 raft_consensus.cc:504] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:41.222970 6710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c98cbfbd0a44335b75b228c217f9132" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:41.223176 6710 raft_consensus.cc:3032] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 LEADER]: Stepping down as leader of term 7
I20240502 21:27:41.223261 6710 raft_consensus.cc:729] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:41.223228 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c98cbfbd0a44335b75b228c217f9132" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:41.223474 6579 raft_consensus.cc:3037] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:41.223543 6710 consensus_queue.cc:260] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:41.223830 6710 raft_consensus.cc:3037] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 7 FOLLOWER]: Advancing to term 8
I20240502 21:27:41.226742 6710 raft_consensus.cc:2396] T 1c98cbfbd0a44335b75b228c217f9132 P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20240502 21:27:41.226778 6579 raft_consensus.cc:2454] T 1c98cbfbd0a44335b75b228c217f9132 P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 8.
I20240502 21:27:41.227314 6774 leader_election.cc:302] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:41.227747 6840 leader_election.cc:288] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 8 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:41.228217 9353 raft_consensus.cc:2781] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Leader election won for term 8
I20240502 21:27:41.228418 9353 raft_consensus.cc:686] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [term 8 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:41.228792 9353 consensus_queue.cc:237] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:41.228983 6840 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ea3b52c673ae4b9f931059c4b0a456fa"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:41.229212 6840 raft_consensus.cc:482] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.229367 6840 raft_consensus.cc:3037] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.232563 6840 raft_consensus.cc:504] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:41.233310 6840 leader_election.cc:288] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 10 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:41.237321 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3b52c673ae4b9f931059c4b0a456fa" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 10 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:41.237545 6968 raft_consensus.cc:3037] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.237684 6840 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "169899da82594d9cb673f8db74764ca2"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:41.237943 6840 raft_consensus.cc:482] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.238078 6840 raft_consensus.cc:3037] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.238804 6710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3b52c673ae4b9f931059c4b0a456fa" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 10 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:41.240165 6968 raft_consensus.cc:2396] T ea3b52c673ae4b9f931059c4b0a456fa P 80a838c00b544bf68ec4cc9c513212b2 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:41.240613 6710 raft_consensus.cc:3032] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 LEADER]: Stepping down as leader of term 9
I20240502 21:27:41.241505 6710 raft_consensus.cc:729] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:41.241533 6840 raft_consensus.cc:504] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:41.242609 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169899da82594d9cb673f8db74764ca2" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:41.242869 6579 raft_consensus.cc:3037] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.243116 6710 consensus_queue.cc:260] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:41.243561 6710 raft_consensus.cc:3037] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.246217 6579 raft_consensus.cc:2454] T 169899da82594d9cb673f8db74764ca2 P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 10.
I20240502 21:27:41.246699 6774 leader_election.cc:302] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters:
I20240502 21:27:41.248667 9077 raft_consensus.cc:2781] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Leader election won for term 10
I20240502 21:27:41.248986 9077 raft_consensus.cc:686] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [term 10 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:41.249323 9077 consensus_queue.cc:237] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:41.251144 6710 raft_consensus.cc:2396] T ea3b52c673ae4b9f931059c4b0a456fa P ab5c055dc24f4944a58d7d2593ddc8ef [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 21:27:41.252565 6840 leader_election.cc:288] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 10 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:41.259367 6840 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8fac59b25c164e658fe71cfdf7ca78cb"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:41.259646 6840 raft_consensus.cc:482] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.259845 6840 raft_consensus.cc:3037] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:41.260890 6439 catalog_manager.cc:5484] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 8 to 10, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 10 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:41.261513 6439 catalog_manager.cc:5484] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 7 to 9, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 9 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:41.261983 6439 catalog_manager.cc:5484] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 9 to 10, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 10 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:41.262444 6439 catalog_manager.cc:5484] T a09d2491d9dc4cbfb20f7adc1087b375 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 9 to 10, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 10 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:41.262907 6439 catalog_manager.cc:5484] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 8 to 10, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 10 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:41.264963 6840 raft_consensus.cc:504] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:41.252900 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169899da82594d9cb673f8db74764ca2" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:41.266512 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fac59b25c164e658fe71cfdf7ca78cb" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:41.266831 6711 raft_consensus.cc:3032] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 LEADER]: Stepping down as leader of term 8
I20240502 21:27:41.266963 6711 raft_consensus.cc:729] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:41.267242 6711 consensus_queue.cc:260] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:41.267992 6711 raft_consensus.cc:3037] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:41.271641 6711 raft_consensus.cc:2396] T 8fac59b25c164e658fe71cfdf7ca78cb P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240502 21:27:41.266839 6709 raft_consensus.cc:3032] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 LEADER]: Stepping down as leader of term 9
I20240502 21:27:41.272074 6709 raft_consensus.cc:729] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ab5c055dc24f4944a58d7d2593ddc8ef, State: Running, Role: LEADER
I20240502 21:27:41.272398 6709 consensus_queue.cc:260] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:41.272737 6709 raft_consensus.cc:3037] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.272958 6776 leader_election.cc:302] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9793130a8e4f45388357b8d2e9f47720; no voters: 80a838c00b544bf68ec4cc9c513212b2, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:41.273438 9248 raft_consensus.cc:2726] T ea3b52c673ae4b9f931059c4b0a456fa P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240502 21:27:41.274684 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fac59b25c164e658fe71cfdf7ca78cb" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:41.274904 6579 raft_consensus.cc:3037] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:41.274964 6840 leader_election.cc:288] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 9 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703)
I20240502 21:27:41.275851 6840 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "093f8fd73ec34d8b9954c1716e910a78"
dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
from {username='slave'} at 127.0.0.1:59334
I20240502 21:27:41.276015 6840 raft_consensus.cc:482] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.276114 6840 raft_consensus.cc:3037] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:41.277261 6579 raft_consensus.cc:2454] T 8fac59b25c164e658fe71cfdf7ca78cb P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 9.
I20240502 21:27:41.278309 6774 leader_election.cc:302] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 9793130a8e4f45388357b8d2e9f47720; no voters: ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:41.278436 6840 raft_consensus.cc:504] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:41.278721 9248 raft_consensus.cc:2781] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Leader election won for term 9
I20240502 21:27:41.278894 9248 raft_consensus.cc:686] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [term 9 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:41.279126 9248 consensus_queue.cc:237] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:41.279393 6579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093f8fd73ec34d8b9954c1716e910a78" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
I20240502 21:27:41.279551 6579 raft_consensus.cc:3037] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:41.279080 6709 raft_consensus.cc:2396] T 169899da82594d9cb673f8db74764ca2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20240502 21:27:41.281837 6579 raft_consensus.cc:2396] T 093f8fd73ec34d8b9954c1716e910a78 P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 9793130a8e4f45388357b8d2e9f47720 for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240502 21:27:41.282713 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093f8fd73ec34d8b9954c1716e910a78" candidate_uuid: "9793130a8e4f45388357b8d2e9f47720" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:41.282951 6968 raft_consensus.cc:3037] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [term 8 FOLLOWER]: Advancing to term 9
I20240502 21:27:41.285389 6968 raft_consensus.cc:2454] T 093f8fd73ec34d8b9954c1716e910a78 P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9793130a8e4f45388357b8d2e9f47720 in term 9.
I20240502 21:27:41.285820 6775 leader_election.cc:302] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 80a838c00b544bf68ec4cc9c513212b2, 9793130a8e4f45388357b8d2e9f47720; no voters: 51bc1a0939fe49348a18d3da18f9feae
I20240502 21:27:41.286087 6840 leader_election.cc:288] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [CANDIDATE]: Term 9 election: Requested vote from peers 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129:38703), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:41.286108 9353 raft_consensus.cc:2781] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Leader election won for term 9
I20240502 21:27:41.286262 9353 raft_consensus.cc:686] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [term 9 LEADER]: Becoming Leader. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:41.286466 9353 consensus_queue.cc:237] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:41.291173 6439 catalog_manager.cc:5484] T fbec2b1b80644a93b9239600157ad957 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 7 to 9, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 9 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:41.292639 6439 catalog_manager.cc:5484] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 9 to 10, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 10 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:41.293388 6439 catalog_manager.cc:5484] T d3093485f30b41939a1380ab7aa9e38b P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 9 to 10, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 10 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:41.296029 6439 catalog_manager.cc:5484] T 169899da82594d9cb673f8db74764ca2 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 9 to 10, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 10 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:41.296964 6439 catalog_manager.cc:5484] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 8 to 10, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 10 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:41.299466 6439 catalog_manager.cc:5484] T 1c98cbfbd0a44335b75b228c217f9132 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 7 to 8, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 8 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:41.307678 8750 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:41.308746 6439 catalog_manager.cc:5484] T 8fac59b25c164e658fe71cfdf7ca78cb P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 8 to 9, leader changed from ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 9 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:41.309530 6439 catalog_manager.cc:5484] T 093f8fd73ec34d8b9954c1716e910a78 P 9793130a8e4f45388357b8d2e9f47720 reported cstate change: term changed from 7 to 9, leader changed from 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129) to 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131). New cstate: current_term: 9 leader_uuid: "9793130a8e4f45388357b8d2e9f47720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:41.334564 6579 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "df917e7196ed4939b75dc52930c87640"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:41.334879 6579 raft_consensus.cc:482] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.335004 6579 raft_consensus.cc:3037] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.339157 6579 raft_consensus.cc:504] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:41.340776 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df917e7196ed4939b75dc52930c87640" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 10 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:41.341065 6709 raft_consensus.cc:3037] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.344269 6709 raft_consensus.cc:2396] T df917e7196ed4939b75dc52930c87640 P ab5c055dc24f4944a58d7d2593ddc8ef [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 10 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240502 21:27:41.344894 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df917e7196ed4939b75dc52930c87640" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 10 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:41.345067 6579 leader_election.cc:288] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 10 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:41.345098 6840 raft_consensus.cc:3032] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 9 LEADER]: Stepping down as leader of term 9
I20240502 21:27:41.345191 6840 raft_consensus.cc:729] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:41.345433 6840 consensus_queue.cc:260] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:41.345808 6840 raft_consensus.cc:3037] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.349797 6840 raft_consensus.cc:2396] T df917e7196ed4939b75dc52930c87640 P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240502 21:27:41.350533 6513 leader_election.cc:302] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae; no voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:41.351048 6579 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:41.351241 6579 raft_consensus.cc:482] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.351372 6579 raft_consensus.cc:3037] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Advancing to term 11
I20240502 21:27:41.356114 6579 raft_consensus.cc:504] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 11 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:41.357798 6579 leader_election.cc:288] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 11 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:41.358145 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:41.358345 6709 raft_consensus.cc:3037] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 10 FOLLOWER]: Advancing to term 11
I20240502 21:27:41.361542 6709 raft_consensus.cc:2396] T 647ab470df6f491a81d040a1647ca95b P ab5c055dc24f4944a58d7d2593ddc8ef [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 11 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20240502 21:27:41.361116 6558 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:41.363747 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3619e8a307de4583a4e470ad583fd7bc"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:41.363934 6577 raft_consensus.cc:482] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.364042 6577 raft_consensus.cc:3037] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Advancing to term 11
I20240502 21:27:41.366533 6577 raft_consensus.cc:504] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 11 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:41.367179 6577 leader_election.cc:288] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 11 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:41.368186 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3619e8a307de4583a4e470ad583fd7bc" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:41.368384 6968 raft_consensus.cc:3037] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 10 FOLLOWER]: Advancing to term 11
I20240502 21:27:41.371440 6968 raft_consensus.cc:2454] T 3619e8a307de4583a4e470ad583fd7bc P 80a838c00b544bf68ec4cc9c513212b2 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 11.
I20240502 21:27:41.371667 8485 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:41.372045 6514 leader_election.cc:302] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2; no voters:
I20240502 21:27:41.392164 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aae0bc634aa7424e935c8bc81a85099b"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:41.392446 6577 raft_consensus.cc:482] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.392577 6577 raft_consensus.cc:3037] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Advancing to term 11
I20240502 21:27:41.393767 9386 raft_consensus.cc:2726] T df917e7196ed4939b75dc52930c87640 P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240502 21:27:41.394114 9386 raft_consensus.cc:2781] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 11 FOLLOWER]: Leader election won for term 11
I20240502 21:27:41.394300 9386 raft_consensus.cc:686] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [term 11 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:41.396598 6577 raft_consensus.cc:504] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 11 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:41.397506 6577 leader_election.cc:288] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 11 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:41.397706 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae0bc634aa7424e935c8bc81a85099b" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:41.397912 6709 raft_consensus.cc:3037] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 10 FOLLOWER]: Advancing to term 11
I20240502 21:27:41.400593 6709 raft_consensus.cc:2454] T aae0bc634aa7424e935c8bc81a85099b P ab5c055dc24f4944a58d7d2593ddc8ef [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 11.
I20240502 21:27:41.401118 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3b6b526ec34c41aea64a2a19f0208f5d"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:41.401291 6577 raft_consensus.cc:482] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.401410 6577 raft_consensus.cc:3037] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.401548 6515 leader_election.cc:302] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:41.397387 9386 consensus_queue.cc:237] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
W20240502 21:27:41.403986 6513 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.226.131:33929, user_credentials={real_user=slave}} blocked reactor thread for 53504.5us
I20240502 21:27:41.404628 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647ab470df6f491a81d040a1647ca95b" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:41.404826 6840 raft_consensus.cc:3032] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 10 LEADER]: Stepping down as leader of term 10
I20240502 21:27:41.404914 6840 raft_consensus.cc:729] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:41.404834 6577 raft_consensus.cc:504] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:41.405189 6840 consensus_queue.cc:260] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:41.405480 6840 raft_consensus.cc:3037] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Advancing to term 11
I20240502 21:27:41.405613 6577 leader_election.cc:288] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 10 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:41.407258 9393 raft_consensus.cc:2781] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 11 FOLLOWER]: Leader election won for term 11
I20240502 21:27:41.407312 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3619e8a307de4583a4e470ad583fd7bc" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:41.407433 9393 raft_consensus.cc:686] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [term 11 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:41.407464 6839 raft_consensus.cc:3032] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 10 LEADER]: Stepping down as leader of term 10
I20240502 21:27:41.407564 6839 raft_consensus.cc:729] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:41.407727 9393 consensus_queue.cc:237] T aae0bc634aa7424e935c8bc81a85099b P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:41.407797 6839 consensus_queue.cc:260] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:41.408088 6839 raft_consensus.cc:3037] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Advancing to term 11
I20240502 21:27:41.408942 6841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6b526ec34c41aea64a2a19f0208f5d" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:41.409127 6841 raft_consensus.cc:3032] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 9 LEADER]: Stepping down as leader of term 9
I20240502 21:27:41.409226 6841 raft_consensus.cc:729] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:41.409442 6841 consensus_queue.cc:260] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:41.409719 6841 raft_consensus.cc:3037] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.411403 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae0bc634aa7424e935c8bc81a85099b" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:41.411664 6968 raft_consensus.cc:3037] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 10 FOLLOWER]: Advancing to term 11
I20240502 21:27:41.412231 6841 raft_consensus.cc:2396] T 3b6b526ec34c41aea64a2a19f0208f5d P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20240502 21:27:41.414697 6968 raft_consensus.cc:2396] T aae0bc634aa7424e935c8bc81a85099b P 80a838c00b544bf68ec4cc9c513212b2 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20240502 21:27:41.416888 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "da14dcc58d684704b3649e85b26fb43c"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:41.417073 6577 raft_consensus.cc:482] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.417168 6577 raft_consensus.cc:3037] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Advancing to term 11
I20240502 21:27:41.419420 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6b526ec34c41aea64a2a19f0208f5d" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:41.419620 6577 raft_consensus.cc:504] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 11 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:41.419598 6968 raft_consensus.cc:3037] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.420145 6577 leader_election.cc:288] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 11 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:41.420596 6690 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:27:41.420930 6840 raft_consensus.cc:2396] T 647ab470df6f491a81d040a1647ca95b P 9793130a8e4f45388357b8d2e9f47720 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20240502 21:27:41.421900 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "66198d766e6e4ceb820bc92bcca67d85"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:41.422055 6577 raft_consensus.cc:482] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.422163 6577 raft_consensus.cc:3037] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Advancing to term 11
I20240502 21:27:41.422493 6839 raft_consensus.cc:2396] T 3619e8a307de4583a4e470ad583fd7bc P 9793130a8e4f45388357b8d2e9f47720 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20240502 21:27:41.422921 6968 raft_consensus.cc:2454] T 3b6b526ec34c41aea64a2a19f0208f5d P 80a838c00b544bf68ec4cc9c513212b2 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 10.
I20240502 21:27:41.423367 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da14dcc58d684704b3649e85b26fb43c" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:41.423525 6514 leader_election.cc:302] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, 80a838c00b544bf68ec4cc9c513212b2; no voters:
I20240502 21:27:41.423573 6839 raft_consensus.cc:3032] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 10 LEADER]: Stepping down as leader of term 10
I20240502 21:27:41.423658 6839 raft_consensus.cc:729] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:41.423905 6839 consensus_queue.cc:260] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:41.424166 6839 raft_consensus.cc:3037] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Advancing to term 11
I20240502 21:27:41.426374 6577 raft_consensus.cc:504] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [term 11 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:41.426612 6839 raft_consensus.cc:2396] T da14dcc58d684704b3649e85b26fb43c P 9793130a8e4f45388357b8d2e9f47720 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20240502 21:27:41.427181 6577 leader_election.cc:288] T 66198d766e6e4ceb820bc92bcca67d85 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 11 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:41.428076 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f0b941baaab542a78748d4d0894cfbf4"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:41.428256 6577 raft_consensus.cc:482] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.428383 6577 raft_consensus.cc:3037] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Advancing to term 11
I20240502 21:27:41.428088 6513 leader_election.cc:302] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae; no voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:41.428649 6949 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:41.430747 6577 raft_consensus.cc:504] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 11 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:41.431366 6577 leader_election.cc:288] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 11 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:41.440088 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:41.441396 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5e38d544ee14490c9fb5d8069440fd6a"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:41.441558 6577 raft_consensus.cc:482] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.441669 6577 raft_consensus.cc:3037] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Advancing to term 11
I20240502 21:27:41.447055 6968 raft_consensus.cc:1261] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 10 FOLLOWER]: Refusing update from remote peer 9793130a8e4f45388357b8d2e9f47720: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20240502 21:27:41.447665 9353 consensus_queue.cc:1035] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:41.447871 9353 consensus_queue.cc:1225] T 6b2b3e30b94d47dd822539cb30f36f81 P 9793130a8e4f45388357b8d2e9f47720 [LEADER]: Peer 80a838c00b544bf68ec4cc9c513212b2 log is divergent from this leader: its last log entry 8.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20240502 21:27:41.447901 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66198d766e6e4ceb820bc92bcca67d85" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:41.448098 6839 raft_consensus.cc:3032] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 10 LEADER]: Stepping down as leader of term 10
I20240502 21:27:41.448189 6839 raft_consensus.cc:729] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:41.448454 6839 consensus_queue.cc:260] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } }
I20240502 21:27:41.448755 6839 raft_consensus.cc:3037] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Advancing to term 11
I20240502 21:27:41.448798 6969 pending_rounds.cc:76] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2: Aborting all ops after (but not including) 5
I20240502 21:27:41.448894 6969 pending_rounds.cc:97] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20240502 21:27:41.448913 9400 raft_consensus.cc:2781] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Leader election won for term 10
I20240502 21:27:41.448968 6969 raft_consensus.cc:2866] T 6b2b3e30b94d47dd822539cb30f36f81 P 80a838c00b544bf68ec4cc9c513212b2 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 21:27:41.449035 9400 raft_consensus.cc:686] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [term 10 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:41.449281 9400 consensus_queue.cc:237] T 3b6b526ec34c41aea64a2a19f0208f5d P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:41.451345 6839 raft_consensus.cc:2396] T 66198d766e6e4ceb820bc92bcca67d85 P 9793130a8e4f45388357b8d2e9f47720 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20240502 21:27:41.452857 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b941baaab542a78748d4d0894cfbf4" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:41.453017 6839 raft_consensus.cc:3032] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 10 LEADER]: Stepping down as leader of term 10
I20240502 21:27:41.453119 6839 raft_consensus.cc:729] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:41.453325 6839 consensus_queue.cc:260] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:41.453593 6839 raft_consensus.cc:3037] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Advancing to term 11
I20240502 21:27:41.456087 6839 raft_consensus.cc:2396] T f0b941baaab542a78748d4d0894cfbf4 P 9793130a8e4f45388357b8d2e9f47720 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 11 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20240502 21:27:41.458741 9396 raft_consensus.cc:2726] T 647ab470df6f491a81d040a1647ca95b P 51bc1a0939fe49348a18d3da18f9feae [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240502 21:27:41.460319 6577 raft_consensus.cc:504] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [term 11 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
W20240502 21:27:41.460700 6515 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.226.130:42515, user_credentials={real_user=slave}} blocked reactor thread for 59135.9us
I20240502 21:27:41.461201 6577 leader_election.cc:288] T 5e38d544ee14490c9fb5d8069440fd6a P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 11 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:41.461501 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da14dcc58d684704b3649e85b26fb43c" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:41.461719 6709 raft_consensus.cc:3037] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 10 FOLLOWER]: Advancing to term 11
I20240502 21:27:41.461891 6710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b941baaab542a78748d4d0894cfbf4" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:41.462188 6710 raft_consensus.cc:3037] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 10 FOLLOWER]: Advancing to term 11
I20240502 21:27:41.465571 6710 raft_consensus.cc:2454] T f0b941baaab542a78748d4d0894cfbf4 P ab5c055dc24f4944a58d7d2593ddc8ef [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 11.
I20240502 21:27:41.465904 6439 catalog_manager.cc:5484] T 3619e8a307de4583a4e470ad583fd7bc P 51bc1a0939fe49348a18d3da18f9feae reported cstate change: term changed from 10 to 11, leader changed from 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131) to 51bc1a0939fe49348a18d3da18f9feae (127.24.226.129). New cstate: current_term: 11 leader_uuid: "51bc1a0939fe49348a18d3da18f9feae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:41.466159 6515 leader_election.cc:302] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters: 9793130a8e4f45388357b8d2e9f47720
I20240502 21:27:41.466711 9386 raft_consensus.cc:2781] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 11 FOLLOWER]: Leader election won for term 11
I20240502 21:27:41.466852 9386 raft_consensus.cc:686] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [term 11 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:41.467141 9386 consensus_queue.cc:237] T f0b941baaab542a78748d4d0894cfbf4 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:41.470561 6709 raft_consensus.cc:2396] T da14dcc58d684704b3649e85b26fb43c P ab5c055dc24f4944a58d7d2593ddc8ef [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 11 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20240502 21:27:41.470841 6577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5b2e443bad344ce89ce69871406c5a29"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:41.471024 6577 raft_consensus.cc:482] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.471134 6577 raft_consensus.cc:3037] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Advancing to term 11
I20240502 21:27:41.472154 6515 leader_election.cc:302] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae; no voters: 9793130a8e4f45388357b8d2e9f47720, ab5c055dc24f4944a58d7d2593ddc8ef
I20240502 21:27:41.472550 9386 raft_consensus.cc:2726] T da14dcc58d684704b3649e85b26fb43c P 51bc1a0939fe49348a18d3da18f9feae [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240502 21:27:41.475178 6577 raft_consensus.cc:504] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [term 11 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:41.476794 8618 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:41.477115 6577 leader_election.cc:288] T 5b2e443bad344ce89ce69871406c5a29 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 11 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
W20240502 21:27:41.477389 6514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.226.132:43031, user_credentials={real_user=slave}} blocked reactor thread for 53855.2us
I20240502 21:27:41.478006 6579 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fb023d72aa8d4248a64d4723e18a3cf6"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:41.478194 6579 raft_consensus.cc:482] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.478297 6579 raft_consensus.cc:3037] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Advancing to term 11
I20240502 21:27:41.480844 6579 raft_consensus.cc:504] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 11 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:41.481216 6820 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 21:27:41.481433 6579 leader_election.cc:288] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 11 election: Requested vote from peers ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515), 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031)
I20240502 21:27:41.482232 6579 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c6d11cba6ca04c17a9f47506824d9aa2"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:41.482396 6579 raft_consensus.cc:482] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.482509 6579 raft_consensus.cc:3037] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Advancing to term 11
I20240502 21:27:41.484920 6579 raft_consensus.cc:504] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [term 11 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:41.485565 6579 leader_election.cc:288] T c6d11cba6ca04c17a9f47506824d9aa2 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 11 election: Requested vote from peers 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:41.486361 6580 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "331bcde3e6bf43c081af836c8cdebdc7"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:41.486521 6580 raft_consensus.cc:482] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.486506 6710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d11cba6ca04c17a9f47506824d9aa2" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 11 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:41.486627 6580 raft_consensus.cc:3037] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.486693 6710 raft_consensus.cc:3037] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 10 FOLLOWER]: Advancing to term 11
I20240502 21:27:41.489132 6580 raft_consensus.cc:504] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:41.489857 6710 raft_consensus.cc:2396] T c6d11cba6ca04c17a9f47506824d9aa2 P ab5c055dc24f4944a58d7d2593ddc8ef [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 11 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240502 21:27:41.489864 6580 leader_election.cc:288] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 10 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), ab5c055dc24f4944a58d7d2593ddc8ef (127.24.226.130:42515)
I20240502 21:27:41.490828 6709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb023d72aa8d4248a64d4723e18a3cf6" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:41.491067 6580 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fbec2b1b80644a93b9239600157ad957"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:41.491344 6580 raft_consensus.cc:482] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:41.491528 6709 raft_consensus.cc:3037] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [term 10 FOLLOWER]: Advancing to term 11
I20240502 21:27:41.492197 6580 raft_consensus.cc:3037] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.492918 6710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331bcde3e6bf43c081af836c8cdebdc7" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef"
I20240502 21:27:41.493125 6710 raft_consensus.cc:3037] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.493717 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2e443bad344ce89ce69871406c5a29" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:41.493916 6840 raft_consensus.cc:3032] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 10 LEADER]: Stepping down as leader of term 10
I20240502 21:27:41.494031 6840 raft_consensus.cc:729] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:41.494290 6840 consensus_queue.cc:260] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:41.494606 6840 raft_consensus.cc:3037] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Advancing to term 11
I20240502 21:27:41.495816 6710 raft_consensus.cc:2454] T 331bcde3e6bf43c081af836c8cdebdc7 P ab5c055dc24f4944a58d7d2593ddc8ef [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 10.
I20240502 21:27:41.497097 6580 raft_consensus.cc:504] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } }
I20240502 21:27:41.497712 6580 leader_election.cc:288] T fbec2b1b80644a93b9239600157ad957 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 10 election: Requested vote from peers 80a838c00b544bf68ec4cc9c513212b2 (127.24.226.132:43031), 9793130a8e4f45388357b8d2e9f47720 (127.24.226.131:33929)
I20240502 21:27:41.497974 6709 raft_consensus.cc:2454] T fb023d72aa8d4248a64d4723e18a3cf6 P ab5c055dc24f4944a58d7d2593ddc8ef [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 11.
I20240502 21:27:41.498116 6840 raft_consensus.cc:2396] T 5b2e443bad344ce89ce69871406c5a29 P 9793130a8e4f45388357b8d2e9f47720 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20240502 21:27:41.498484 6515 leader_election.cc:302] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:41.498804 6515 leader_election.cc:302] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bc1a0939fe49348a18d3da18f9feae, ab5c055dc24f4944a58d7d2593ddc8ef; no voters:
I20240502 21:27:41.498878 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d11cba6ca04c17a9f47506824d9aa2" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 11 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:41.498836 6579 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1c98cbfbd0a44335b75b228c217f9132"
dest_uuid: "51bc1a0939fe49348a18d3da18f9feae"
from {username='slave'} at 127.0.0.1:41904
I20240502 21:27:41.499032 6840 raft_consensus.cc:3032] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 10 LEADER]: Stepping down as leader of term 10
I20240502 21:27:41.499145 6840 raft_consensus.cc:729] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:41.499044 9386 raft_consensus.cc:2781] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 10 FOLLOWER]: Leader election won for term 10
I20240502 21:27:41.499359 9386 raft_consensus.cc:686] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [term 10 LEADER]: Becoming Leader. State: Replica: 51bc1a0939fe49348a18d3da18f9feae, State: Running, Role: LEADER
I20240502 21:27:41.499398 6840 consensus_queue.cc:260] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:41.499542 9396 raft_consensus.cc:2781] T fb023d72aa8d4248a64d4723e18a3cf6 P 51bc1a0939fe49348a18d3da18f9feae [term 11 FOLLOWER]: Leader election won for term 11
I20240502 21:27:41.499658 6968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66198d766e6e4ceb820bc92bcca67d85" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:41.499755 6840 raft_consensus.cc:3037] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Advancing to term 11
I20240502 21:27:41.499881 6968 raft_consensus.cc:3037] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 10 FOLLOWER]: Advancing to term 11
I20240502 21:27:41.500146 9386 consensus_queue.cc:237] T 331bcde3e6bf43c081af836c8cdebdc7 P 51bc1a0939fe49348a18d3da18f9feae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "ab5c055dc24f4944a58d7d2593ddc8ef" member_type: VOTER last_known_addr { host: "127.24.226.130" port: 42515 } }
I20240502 21:27:41.500660 6967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb023d72aa8d4248a64d4723e18a3cf6" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:41.500856 6967 raft_consensus.cc:3037] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 10 FOLLOWER]: Advancing to term 11
I20240502 21:27:41.502647 6840 raft_consensus.cc:2396] T c6d11cba6ca04c17a9f47506824d9aa2 P 9793130a8e4f45388357b8d2e9f47720 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240502 21:27:41.502789 6968 raft_consensus.cc:2454] T 66198d766e6e4ceb820bc92bcca67d85 P 80a838c00b544bf68ec4cc9c513212b2 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 11.
I20240502 21:27:41.503378 6970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e38d544ee14490c9fb5d8069440fd6a" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:41.503562 6970 raft_consensus.cc:3037] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 10 FOLLOWER]: Advancing to term 11
I20240502 21:27:41.503587 6967 raft_consensus.cc:2396] T fb023d72aa8d4248a64d4723e18a3cf6 P 80a838c00b544bf68ec4cc9c513212b2 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20240502 21:27:41.503945 6965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec2b1b80644a93b9239600157ad957" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:41.504211 6965 raft_consensus.cc:3037] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.506261 6970 raft_consensus.cc:2454] T 5e38d544ee14490c9fb5d8069440fd6a P 80a838c00b544bf68ec4cc9c513212b2 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 11.
I20240502 21:27:41.506579 6971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2e443bad344ce89ce69871406c5a29" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:41.506742 6971 raft_consensus.cc:3037] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 10 FOLLOWER]: Advancing to term 11
I20240502 21:27:41.506877 6965 raft_consensus.cc:2454] T fbec2b1b80644a93b9239600157ad957 P 80a838c00b544bf68ec4cc9c513212b2 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 10.
I20240502 21:27:41.508800 6965 raft_consensus.cc:1261] T c9e728880af44303a975a63a4bd7bc62 P 80a838c00b544bf68ec4cc9c513212b2 [term 10 FOLLOWER]: Refusing update from remote peer ab5c055dc24f4944a58d7d2593ddc8ef: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20240502 21:27:41.509279 6971 raft_consensus.cc:2454] T 5b2e443bad344ce89ce69871406c5a29 P 80a838c00b544bf68ec4cc9c513212b2 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bc1a0939fe49348a18d3da18f9feae in term 11.
I20240502 21:27:41.509413 6967 raft_consensus.cc:1261] T 875f83053fee45b3bc1921b24745e9da P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Refusing update from remote peer ab5c055dc24f4944a58d7d2593ddc8ef: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20240502 21:27:41.509649 6966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331bcde3e6bf43c081af836c8cdebdc7" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "80a838c00b544bf68ec4cc9c513212b2"
I20240502 21:27:41.509847 6966 raft_consensus.cc:3037] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 9 FOLLOWER]: Advancing to term 10
I20240502 21:27:41.509919 9314 consensus_queue.cc:1035] T c9e728880af44303a975a63a4bd7bc62 P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:41.510327 9314 consensus_queue.cc:1035] T 875f83053fee45b3bc1921b24745e9da P ab5c055dc24f4944a58d7d2593ddc8ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:41.510720 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e38d544ee14490c9fb5d8069440fd6a" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"
I20240502 21:27:41.510890 6839 raft_consensus.cc:3032] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 10 LEADER]: Stepping down as leader of term 10
I20240502 21:27:41.511001 6839 raft_consensus.cc:729] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 9793130a8e4f45388357b8d2e9f47720, State: Running, Role: LEADER
I20240502 21:27:41.512365 6966 raft_consensus.cc:2396] T 331bcde3e6bf43c081af836c8cdebdc7 P 80a838c00b544bf68ec4cc9c513212b2 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 51bc1a0939fe49348a18d3da18f9feae for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20240502 21:27:41.512368 6839 consensus_queue.cc:260] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9793130a8e4f45388357b8d2e9f47720" member_type: VOTER last_known_addr { host: "127.24.226.131" port: 33929 } } peers { permanent_uuid: "51bc1a0939fe49348a18d3da18f9feae" member_type: VOTER last_known_addr { host: "127.24.226.129" port: 38703 } } peers { permanent_uuid: "80a838c00b544bf68ec4cc9c513212b2" member_type: VOTER last_known_addr { host: "127.24.226.132" port: 43031 } }
I20240502 21:27:41.512665 6839 raft_consensus.cc:3037] T 5e38d544ee14490c9fb5d8069440fd6a P 9793130a8e4f45388357b8d2e9f47720 [term 10 FOLLOWER]: Advancing to term 11
I20240502 21:27:41.512087 6840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbec2b1b80644a93b9239600157ad957" candidate_uuid: "51bc1a0939fe49348a18d3da18f9feae" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "9793130a8e4f45388357b8d2e9f47720"